{ "source_file": "0930", "source_platform": "Medium", "author": "Cristian Leo", "title_listed": "Building a Knowledge Graph From Scratch Using LLMs", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/building-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07/", "publish_date": "2024-11-25", "corpus_stage": "", "notes": "", "id": "0852990cc19f", "collected_at": "2026-10-02T05:58:31", "title": "Building a Knowledge Graph From Scratch Using LLMs | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Building a Knowledge Graph From Scratch Using LLMs\nTurn your Pandas Data Frame into a Knowledge Graph using LLMs. Build your own LLM graph-builder and QA your KG.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nNovember 25, 202439 min read\n### Turn your Pandas data frame into a knowledge graph using LLMs. Build your own LLM graph-builder from scratch, implement LLMGraphTransformer by LangChain, and QA your KG.\n![Knowledge Graph about 1000 Movies from Wikipedia - Image by Author](../images/000.png)\nKnowledge Graph about 1000 Movies from Wikipedia - Image by Author\nIn today's AI world, knowledge graphs are becoming increasingly important, as they enable many of the knowledge retrieval systems behind LLMs. Many data science teams across companies are investing heavily in retrieval augmented generation (RAG), as it's an efficient way to improve the output accuracy of LLMs and prevent hallucinations.\nBut there is more to it; on a personal note, graph-RAG is democratizing the AI space. This is because before if we wanted to customize a model to a use-case - either for fun or business - we would have three options: pre-training the model providing a bigger exposure to a dataset within the industry of your use-case, fine-tuning the model on a specific dataset, and context prompting.\nAs for pre-training, this option is incredibly expensive and technical, and it's not an option for most developers.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\nFine-tuning is easier than pre-training, and although the cost of fine-tuning depends on the model and the training corpus, it's generally a more affordable option. This one was the go-to strategy for most of the AI developers. However, new models are released every week and you would need to fine-tune a new model constantly.\nThe third option involves providing the knowledge directly in the prompt. However, this works only when the knowledge required for the model to learn is relatively small. Even though the context for the models is getting bigger and bigger, the accuracy of recalling an element is inversely correlated to the size of the context provided.\nNeither of the three options sounds like the right one. Is there another option for the model to learn all the knowledge required to be specialized in a certain task or topic? No.\nBut, the model doesn't need to learn all the knowledge at once, as when we interrogate the LLM we are likely trying to get one or a few pieces of information. Here comes graph-RAG to help, by providing an informational retrieval that based on the query retrieves the information needed, without requiring any further training.\nLet's take a look at what a **Graph-RAG** looks like:\n * **Graph Building** : here, we create nodes (entities) and edges (relationships) from a data source and load them into our KG. This is usually a more manual step where we use a query language, often OpenCypher to upload both entities and connect them to each other via edges.\n * **Node indexing** : this step involves creating a data structure that will allow us to retrieve the data efficiently. In knowledge graphs, this typically involves creating a vector search index, where every index is associated with a vector embedding.\n * **Graph retriever** : here, we build a retrieval function that will allow us to compute a similarity score and retrieve the most relevant nodes that will be used as context for the LLM to provide an answer. In the simplest case, we compute the cosine similarity over the query, which is converted to a vector embedding, and all the vector embeddings in the vector index.\n * **RAG Evaluation** : the last step is useful to actually measure the accuracy and performance of the LLM. It's useful during experimentation to perform how different LLMs, and RAG frameworks perform on your specific use case.\n\n\nNow that we have an overall view of what a RAG pipeline consists of, we may be attracted to just jump and experiment with complex math functions for graph retrieval to guarantee the best information retrieval accuracy possible. But ... hold on. We don't have a knowledge graph yet. This step may seem like the classic data cleaning and preprocessing step in data science (boring...). But what if I tell you there is a better alternative? An option that introduces more science and automation. Indeed, recent studies are focusing on how to automate the construction of a Knowledge Graph, since this step is key for good information retrieval. Just think about it, if the data in the KG is not good, there is no way your graph-RAG will have a state-of-the-art performance.\nIn this article, we will delve into the first step: How to build a knowledge graph without actually building it.\n## From CSV to Knowledge Graph\nNow, let's go over a practical example to make things more concrete. Let's tackle one of the most important existential issues: what movie to watch? ... How many times have you been bored, and tired from work and the only thing you could do was watch a movie? You start scrolling among movies until you realize that two hours have passed by.\nTo solve this issue, let's create a knowledge graph using a [Wikipedia Movie Dataset](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots), and chat with the KG. Firstly, let's implement a \"from scratch\" solution using LLMs. Then, let's look at one of the latest implementations via LangChain (still in the experimental phase as of November 2024), one of the most popular and powerful LLM frameworks available, and another popular solution with LlamaIndex.\nLet's download this public dataset from Kaggle ([License: CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)):\n> [**Wikipedia Movie Plots**](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots)\nOr if you are lazy, just go ahead and clone my GitHub repo:\n> [**GitHub - cristianleoo/rag-knowledge-graph**](https://github.com/cristianleoo/rag-knowledge-graph)\nThe folder `knowledge-builder` contains both the Jupyter Notebook and the data we will be covering in this article.\n### Prerequisites\nBefore we can start, we need to have access to Neo4j Desktop and an LLM API Key or a local LLM. If you already have them, feel free to skip this section and jump to the action. If not, let's set them up, and don't worry it will be completely free.\nThere are several ways to leverage Neo4j, but for the sake of simplicity we will use Neo4j desktop, hence we will host the database locally. But it's a small dataset, so you won't be destroying your laptop by running this application.\nTo install Neo4j simply visit the [Neo4j Desktop Download Page](https://neo4j.com/download/?utm_source=GSearch&utm_medium=PaidSearch&utm_campaign=Evergreen&utm_content=AMS-Search-SEMCE-DSA-None-SEM-SEM-NonABM&utm_term=&utm_adgroup=DSA&gad_source=1&gclid=CjwKCAiAxea5BhBeEiwAh4t5K2poPd-WtFHvrB6_1SD4ON3P9iVZ4SIZSqrdqDDWPsc7A5bqGwFiVBoCPfcQAvD_BwE) and click Download. Open **Neo4j Desktop** after installation. Sign in or create a **Neo4j account** (required to activate the software).\nOnce logged in, create a **New Project** :\n * Click on the `**+**`button in the top-left corner.\n * Name your project (e.g., \"Wiki Movie KG\").\n\n\nInside your project, click on `**Add Database**`. Select**Local DBMS** and click **Create a Local Graph**.\nConfigure your database:\n * **Name** : Enter a name (e.g., `neo4j`).\n * **Password** : Set a password (e.g., `ilovemovies`). Remember this password for later.\n\n\nClick **Create** to initialize the database.\nNext, let's move to our LLM. The preferred way of running this notebook is using Ollama. **Ollama** is a locally hosted LLM solution that lets you download and set up LLMs on your laptop really really easily. It supports many open-source LLMs including Llama by Meta and Gemma by Google. I prefer this step as running your local LLM is free (excluding the degradation/energy cost of your laptop), and private, and it's just more exciting.\nTo download Ollama visit [Ollama's official website](https://ollama.ai/), and download the installer for your operating system. Open the **Ollama** application after installation.\nOpen a terminal and use the following command to list available models:\nbash\n\n```\nollama list\n```\n\nInstall and run a model. we will be using `qwen2.5-coder:latest` , which is a 7B Language Model fine-tuned on code tasks.\nbash\n\n```\nollama run qwen2.5-coder:latest\n```\n\nVerify the installation:\nbash\n\n```\nollama list\n```\n\nYou should now see:\nbash\n\n```\nqwen2.5-coder:latest\n```\n\nAnother free alternative is Gemini by Google, which lets us run 1500 requests per day. This solution actually outperforms the previous one since we are using a bigger and more powerful model. However, you may hit the limit depending on how many times you execute the script in a day.\nTo get a free API Key with Gemini, visit the [website](https://www.googleadservices.com/pagead/aclk?sa=L&ai=DChcSEwiL4OC_neSJAxV-cEcBHSPmEjUYABAAGgJxdQ&co=1&ase=2&gclid=CjwKCAiAxea5BhBeEiwAh4t5K-5Er62K62POougv5RZFpAglDFSVg43SrYLQM6MWWPP9LuJpTrnjTRoCNo4QAvD_BwE&ei=SlU6Z8KoOdWt5NoP0oio6QY&ohost=www.google.com&cid=CAESVuD2RgfXSVKvkdgIqHdVxFkchLNjCfya9TRqTM6SsW3mXjsq4CFoeiUD5rFUM47GMTu1RN9W0JE9DzrlzBDI75iT7oDTASAPGQU8SpT5_Z6M101LwjzM&sig=AOD64_3PJYSJ7hFfGD_flW0QfFen0Mypnw&q&sqi=2&nis=4&adurl&ved=2ahUKEwjC9dq_neSJAxXVFlkFHVIEKm0Q0Qx6BAgLEAE) and click \"Get an API Key\". Then follow the instructions and copy the API key generated. We will use it in a moment.\n### Graph Builder from Scratch\nLet's start by importing a few libraries required for the project:\npython\n\n```\n# Type hints\nfrom typing import Any, Dict, List, Tuple\n\n# Standard library\nimport ast\nimport logging\nimport re\nimport warnings\n\n# Third-party packages - Data manipulation\nimport pandas as pd\nfrom tqdm import tqdm\n\n# Third-party packages - Environment & Database\nfrom dotenv import load_dotenv\nfrom neo4j import GraphDatabase\n\n# Third-party packages - Error handling & Retry logic\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\n# Langchain - Core\nfrom langchain.chains import GraphCypherQAChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain_core.documents import Document\n\n# Langchain - Models & Connectors\nfrom langchain_google_genai import ChatGoogleGenerativeAI, GoogleGenerativeAI\nfrom langchain_ollama.llms import OllamaLLM\n\n# Langchain - Graph & Experimental\nfrom langchain_community.graphs import Neo4jGraph\nfrom langchain_experimental.graph_transformers import LLMGraphTransformer\n\n# Suppress warnings\nwarnings.filterwarnings('ignore')\n\n# Load environment variables\nload_dotenv()\n```\n\nAs you can see, LangChain doesn't do a good job with the code organization, resulting in quite a few lines of import. Let's break down the libraries we are importing:\n 1. `**os**`**and**`**dotenv**`: Help us manage environment variables (like database credentials).\n 2. `**pandas**`: Used to handle and process the movie dataset.\n 3. `**neo4j**`: This library connects Python to the Neo4j graph database.\n 4. `**langchain**`: Provides tools to work with Language Models (LLMs) and graphs.\n 5. `**tqdm**`: Adds a nice UI to print statements. We will use it to show a progress bar in loops, so we know how much processing is left.\n 6. `**warnings**`: Suppresses unnecessary warnings for a cleaner output.\n\n\nWe load the movie dataset, which contains information about 34,886 movies from around the world. The dataset is publicly available on [Kaggle](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots) ([License: CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)). However, if you cloned my GitHub repo, the dataset will already be present in the data folder:\npython\n\n```\nmovies = pd.read_csv('data/wiki_movies.csv') # adjust the path if you manually downloaded the dataset\nmovies.head()\n```\n\nHere, we can see the following features:\n * _Release Year:_ The year in which the movie was released\n * _Title:_ The movie title\n * _Origin/Ethnicity: The o_rigin of the movie (i.e. American, Bollywood, Tamil, etc.)\n * _Director:_ Director(s)\n * _Plot:_ Main actor and actresses\n * _Genre:_ Movie Genre(s)\n * _Wiki Page:_ URL of the Wikipedia page from which the plot description was scraped\n * _Plot:_ Long-form description of movie plot\n\n\nBy looking at these features, we could quickly come up with some of the labels and relationships we would like to see in our KG. Since this is a movie dataset, a movie would be one of them. Moreover, we may be interested in querying for specific actors and directors. Hence, we end up with three labels for our nodes: Movie, Actor, and Director. Of course, we could include more labels. However, let's stop here for the sake of simplicity.\nFor the sake of simplicity, let's clean a little bit this dataset, and extract only the first 1000 rows:\npython\n\n```\ndef clean_data(df: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Clean and preprocess DataFrame.\n\n Args:\n data: Input DataFrame\n\n Returns:\n Cleaned DataFrame\n \"\"\"\n df.drop([\"Wiki Page\"], axis=1, inplace=True)\n\n # Drop duplicates\n df = df.drop_duplicates(subset='Title', keep='first')\n\n # Get object columns\n col_obj = df.select_dtypes(include=[\"object\"]).columns\n\n # Clean string columns\n for col in col_obj:\n # Strip whitespace\n df[col] = df[col].str.strip()\n\n # Replace unknown/empty values\n df[col] = df[col].apply(\n lambda x: None if pd.isna(x) or x.lower() in [\"\", \"unknown\"] \n else x.capitalize()\n )\n\n # Drop rows with any null values\n df = df.dropna(how=\"any\", axis=0)\n\n return df\n\nmovies = clean_data(movies).head(1000)\nmovies.head()\n```\n\nHere, we are dropping the `Wiki Page` column, which contains the link to the Wikipedia page. However, feel free to keep it, as this could be a property for the `Movie` nodes. Next, we drop all the duplicates by title, and we clean all the _string_ (`object`) columns. Lastly, we keep only the first 1000 movies.\nSince our knowledge graph will be hosted on Neo4j, let's set up a helper class to establish the connection and provide useful methods:\npython\n\n```\nclass Neo4jConnection:\n def __init__(self, uri, user, password):\n self.driver = GraphDatabase.driver(uri, auth=(user, password))\n\n def close(self):\n self.driver.close()\n print(\"Connection closed\")\n\n def reset_database(self):\n with self.driver.session() as session:\n session.run(\"MATCH (n) DETACH DELETE n\")\n print(\"Database resetted successfully!\")\n\n def execute_query(self, query, parameters=None):\n with self.driver.session() as session:\n result = session.run(query, parameters or {})\n return [record for record in result]\n```\n\nIn the initialization (`__init__`), we set up the connection to the Neo4j database using the database URL (`uri`), username, and password. We will pass these variables later when we initialize the class.\nThe method`**close**`terminates the connection to the database.\n`**reset_database**`deletes all nodes and relationships in the database using the Cypher command`MATCH (n) DETACH DELETE n`.\n`**execute_query**`runs a given query (like adding a movie or fetching relationships) and returns the results.\nNext, let's connect to the database using the helper class:\npython\n\n```\nuri = \"bolt://localhost:7687\"\nuser = \"neo4j\"\npassword = \"your_password_here\"\nconn = Neo4jConnection(uri, user, password)\nconn.reset_database()\n```\n\nBy default `uri`, and `user` will match the ones provided above. As for `password` , this will be the one you defined while creating the database. Moreover, let's `**reset_database**`to ensure we start with a clean slate by removing any existing data.\nIf you encounter any errors related to APOC not being installed in your database, go to Neo4j -> click on the database -> Plugins -> Install APOC:\n![Neo4j Desktop - Image by Author](../images/001.png)\nNeo4j Desktop - Image by Author\nWe now need to take each movie from the dataset and turn it into a node in our graph. In this section, we will do it manually, whereas in the next sections, we will leverage an LLM to do it for us.\npython\n\n```\ndef parse_number(value: Any, target_type: type) -> Optional[float]:\n \"\"\"Parse string to number with proper error handling.\"\"\"\n if pd.isna(value):\n return None\n try:\n cleaned = str(value).strip().replace(',', '')\n return target_type(cleaned)\n except (ValueError, TypeError):\n return None\n\ndef clean_text(text: str) -> str:\n \"\"\"Clean and normalize text fields.\"\"\"\n if pd.isna(text):\n return \"\"\n return str(text).strip().title()\n```\n\nLet's create two short functions - `**parse_number**`**and**`**clean_text**`- to convert data into numbers for numerical columns, and properly format the text columns. If the conversion fails (e.g., if the value is empty), they return`None` in case of a numerical column, and an empty string for object columns.\nNext, let's create a to iteratively load data into our KG:\nphp\n\n```\ndef load_movies_to_neo4j(movies_df: pd.DataFrame, connection: GraphDatabase) -> None:\n \"\"\"Load movie data into Neo4j with progress tracking and error handling.\"\"\"\n\n logger = logging.getLogger(__name__)\n logger.setLevel(logging.INFO)\n\n # Query templates\n MOVIE_QUERY = \"\"\"\n MERGE (movie:Movie {title: $title})\n SET movie.year = $year,\n movie.origin = $origin,\n movie.genre = $genre,\n movie.plot = $plot\n \"\"\"\n\n DIRECTOR_QUERY = \"\"\"\n MATCH (movie:Movie {title: $title})\n MERGE (director:Director {name: $name})\n MERGE (director)-[:DIRECTED]->(movie)\n \"\"\"\n\n ACTOR_QUERY = \"\"\"\n MATCH (movie:Movie {title: $title})\n MERGE (actor:Actor {name: $name})\n MERGE (actor)-[:ACTED_IN]->(movie)\n \"\"\"\n\n # Process each movie\n for _, row in tqdm(movies_df.iterrows(), total=len(movies_df), desc=\"Loading movies\"):\n try:\n # Prepare movie parameters\n movie_params = {\n \"title\": clean_text(row[\"Title\"]),\n \"year\": parse_number(row[\"Release Year\"], int),\n \"origin\": clean_text(row[\"Origin/Ethnicity\"]),\n \"genre\": clean_text(row[\"Genre\"]),\n \"plot\": str(row[\"Plot\"]).strip()\n }\n\n # Create movie node\n connection.execute_query(MOVIE_QUERY, parameters=movie_params)\n\n # Process directors\n for director in str(row[\"Director\"]).split(\" and \"):\n director_params = {\n \"name\": clean_text(director),\n \"title\": movie_params[\"title\"]\n }\n connection.execute_query(DIRECTOR_QUERY, parameters=director_params)\n\n # Process cast\n if pd.notna(row[\"Cast\"]):\n for actor in row[\"Cast\"].split(\",\"):\n actor_params = {\n \"name\": clean_text(actor),\n \"title\": movie_params[\"title\"]\n }\n connection.execute_query(ACTOR_QUERY, parameters=actor_params)\n\n except Exception as e:\n logger.error(f\"Error loading {row['Title']}: {str(e)}\")\n continue\n\n logger.info(\"Finished loading movies to Neo4j\")\n```\n\nTwo important keywords to know to understand the Cypher query above are `MERGE` and `SET`.\n`**MERGE**`ensures the node or relationship exists; if not, it creates it. Therefore, it combines both the`MATCH` , and `CREATE` clauses, where `MATCH` allows us to search within the graph for a certain structure, and `CREATE` to create nodes and relationships. Therefore, `MERGE` first check if the node/edge we are creating doesn't exist, if it doesn't then it creates it.\nIn the function above we use `MERGE` to create a node for every Movie, Director, and Actor. In particular, since we have features for actors (Star1, Star2, Star3, and Star4), we create an Actor node for each column.\nNext, we create two relationships: one from Director to Movie (`DIRECTED`), and from Actor to Movie (`ACTED_IN`).\n`**SET**`is used to update node or edge properties. In this case, we are providing the Movie node with Year, Rating, Genre, Runtime, and Overview; and the Director, and Actor with a Name property.\nAlso, note that we are using the special symbol `$` to define a parameter.\nNext, let's call the function and load all the movies:\npython\n\n```\nload_movies_to_neo4j(movies, conn)\n```\n\nIt will take approximately a minute to load all 1000 movies.\nOnce, the execution is completed, let's run a Cypher query to check if the movies are uploaded correctly：\npython\n\n```\nquery = \"\"\"\nMATCH (m:Movie)-[:ACTED_IN]-(a:Actor)\nRETURN m.title, a.name\nLIMIT 10;\n\"\"\"\nresults = conn.execute_query(query)\nfor record in results:\n print(record)\n```\n\nThis query should now look familiar. We used `**MATCH**`to find patterns in the graph.\n * `(m:Movie)`: Matches nodes labeled \"Movie\".\n * `[:ACTED_IN]`: Matches relationships of type \"ACTED_IN\".\n * `(a:Actor)`: Matches nodes labeled \"Actor\".\n\n\nAlso, we used `**RETURN**`to specify what to display—in this case, movie titles and actor names, and`**LIMIT**`to restrict the result to the first 10 matches.\nYou should get an output similar to this:\nbash\n\n```\n[,\n ,\n ,\n ,\n ,\n ,\n ,\n ,\n ,\n ]\n```\n\nOk, this list of 10 records representing movies and actors confirms that the movies have been uploaded on the KG.\nNext, let's see the actual graph. Switch to Neo4j Desktop, select the Movie database you created for this exercise, and click open with Neo4j Browser. This will open a new tab where you will able to run Cypher queries. Then, run the following query:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nYou should now see something like this:\n![Manual Knowledge Graph - Image by Author](../images/002.jpeg)\nManual Knowledge Graph - Image by Author\nQuite cool right?\nHowever, this did require some time to explore the dataset, do some cleaning, and manually write the Cypher queries. But come on, it's the Chat GPT era, of course, we don't need to do that anymore unless you want it. Multiple approaches are showing several ways to automate this process. In the next section, we will create a basic one leveraging an LLM.\n### Creating a Custom LLM to Automate Data Upload\nIn this section, we create a custom process where an LLM automatically generates node definitions, relationships, and Cypher queries based on the dataset. This approach could be used on other Dataframes as well as automatically recognize the schema. However, consider that it won't match the performances of modern solutions, like LLMGraphTransformer from LangChain, which we will cover in the next section. Instead, use this section to understand a possible \"from-scratch\" workflow, to get creative, and later design your own `Graph-Builder`. Indeed, if there is one main limitation of the current **SOTA** (_State-Of-The-Art_) approaches, that's being highly sensitive to the nature of data, and patterns. Therefore, being able to think outside the box is incredibly important in order to either design your Graph-RAG framework from scratch or being able to adapt an existing SOTA Graph-RAG to your needs.\nNow, let's get into it, and set up the LLM we will be using for this exercise. You could use any LLM supported by LangChain for this exercise, as long as it's enough powerful to match decent performances.\nTwo free approaches are Gemini, which is free up to 1500 requests per day using their `Gemini Flash` model, and `Ollama`, which lets you easily download open-source models on your laptop and set up an API that you can easily call using LangChain. I tested the notebook with both Gemini, and custom Ollama models, and although Gemini guarantees superior performances, I would highly recommend going with Ollama for learning purposes, as playing around with \"your own\" LLM is just more cool.\nIn the Ollama example, we will be using `qwen2.5-coder 7B`, which is fine-tune on Code-specific tasks, and has impressive performance on code generation, reasoning, and fixing. Based on your memory availability, and laptop performance, you could download the 14B, or 32B version, which would guarantee higher performance.\nLet's initialize the model:\npython\n\n```\n# llm = GoogleGenerativeAI(model=\"gemini-1.5-flash\", google_api_key=api_key) # if you are using Google API\nllm = OllamaLLM(model=\"qwen2.5-coder:latest\")\n```\n\nIf you chose Gemini as your solution, uncomment the first line of code, and comment on the second one. Also, if you chose Gemini, remember to provide the API Key.\nLet's start by extracting the structure of the dataset and defining the nodes and their properties:\npython\n\n```\nnode_structure = \"n\".join([\n f\"{col}: {', '.join(map(str, movies[col].unique()[:3]))}...\" \n for col in movies.columns\n])\nprint(node_structure)\n```\n\nFor each column in the dataset (e.g., `Genre`, `Director`), we display a few sample values. This gives the LLM an understanding of the data format and typical values for each column.\nbash\n\n```\nRelease Year: 1907, 1908, 1909...\nTitle: Daniel boone, Laughing gas, The adventures of dollie...\nOrigin/Ethnicity: American...\nDirector: Wallace mccutcheon and ediwin s. porter, Edwin stanton porter, D. w. griffith...\nCast: William craven, florence lawrence, Bertha regustus, edward boulden, Arthur v. johnson, linda arvidson...\nGenre: Biographical, Comedy, Drama...\nPlot: Boone's daughter befriends an indian maiden as boone and his companion start out on a hunting expedition. while he is away, boone's cabin is attacked by the indians, who set it on fire and abduct boone's daughter. boone returns, swears vengeance, then heads out on the trail to the indian camp. his daughter escapes but is chased. the indians encounter boone, which sets off a huge fight on the edge of a cliff. a burning arrow gets shot into the indian camp. boone gets tied to the stake and tortured. the burning arrow sets the indian camp on fire, causing panic. boone is rescued by his horse, and boone has a knife fight in which he kills the indian chief.[2], The plot is that of a black woman going to the dentist for a toothache and being given laughing gas. on her way walking home, and in other situations, she can't stop laughing, and everyone she meets \"catches\" the laughter from her, including a vendor and police officers., On a beautiful summer day a father and mother take their daughter dollie on an outing to the river. the mother refuses to buy a gypsy's wares. the gypsy tries to rob the mother, but the father drives him off. the gypsy returns to the camp and devises a plan. they return and kidnap dollie while her parents are distracted. a rescue crew is organized, but the gypsy takes dollie to his camp. they gag dollie and hide her in a barrel before the rescue party gets to the camp. once they leave the gypsies and escapes in their wagon. as the wagon crosses the river, the barrel falls into the water. still sealed in the barrel, dollie is swept downstream in dangerous currents. a boy who is fishing in the river finds the barrel, and dollie is reunited safely with her parents...\n```\n\n**Generating Nodes** Next, we use an L**LM prompt template** to instruct the model on how to extract nodes and their properties. Let's first take a look at what the whole code looks like:\npython\n\n```\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\ndef validate_node_definition(node_def: Dict) -> bool:\n \"\"\"Validate node definition structure\"\"\"\n if not isinstance(node_def, dict):\n return False\n return all(\n isinstance(v, dict) and all(isinstance(k, str) for k in v.keys())\n for v in node_def.values()\n )\n\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef get_node_definitions(chain, structure: str, example: Dict) -> Dict[str, Dict[str, str]]:\n \"\"\"Get node definitions with retry logic\"\"\"\n try:\n # Get response from LLM\n response = chain.invoke({\"structure\": structure, \"example\": example})\n\n # Parse response\n node_defs = ast.literal_eval(response)\n\n # Validate structure\n if not validate_node_definition(node_defs):\n raise ValueError(\"Invalid node definition structure\")\n\n return node_defs\n\n except (ValueError, SyntaxError) as e:\n logger.error(f\"Error parsing node definitions: {e}\")\n raise\n\n# Updated node definition template\nnode_example = {\n \"NodeLabel1\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel2\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel3\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n}\n\ndefine_nodes_prompt = PromptTemplate(\n input_variables=[\"example\", \"structure\"],\n template=(\"\"\"\n Analyze the dataset structure below and extract the entity labels for nodes and their properties.n\n The node properties should be based on the dataset columns and their values.n\n Return the result as a dictionary where the keys are the node labels and the values are the node properties.nn\n Example: {example}nn\n\n Dataset Structure:n{structure}nn\n\n Make sure to include all the possible node labels and their properties.n\n If a property can be its own node, include it as a separate node label.n\n Please do not report triple backticks to identify a code block, just return the list of tuples.n\n Return only the dictionary containing node labels and properties, and don't include any other text or quotation.\n\n \"\"\"\n ),\n)\n\n# Execute with error handling\ntry:\n node_chain = define_nodes_prompt | llm\n\n node_definitions = get_node_definitions(node_chain, structure=node_structure, example=node_example)\n logger.info(f\"Node Definitions: {node_definitions}\")\nexcept Exception as e:\n logger.error(f\"Failed to get node definitions: {e}\")\n raise\n```\n\nHere, we first set up logging using the `logging` library, which is a Python module to track events during execution (like errors or status updates):\npython\n\n```\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n```\n\nWe use `**basicConfig**`to configure logging to display messages of level`INFO` or higher, and initiate the `**logger**`instance, which we will use throughout the code to log messages.\nThis step is not really required, and you could replace it with just print statements. However, it's a good engineering practice.\nNext, we create a function to validate the nodes generated by the LLM:\npython\n\n```\ndef validate_node_definition(node_def: Dict) -> bool:\n \"\"\"Validate node definition structure\"\"\"\n if not isinstance(node_def, dict):\n return False\n return all(\n isinstance(v, dict) and all(isinstance(k, str) for k in v.keys())\n for v in node_def.values()\n )\n```\n\nThe input of the function is a dictionary, where keys are node labels (e.g., `Movie`) and values are dictionaries of properties (e.g., `title`, `year`).\nFirst, the function checks if `node_def` is a dictionary, and verifies that each value in the dictionary is also a dictionary and that all keys inside these dictionaries are strings. Then it returns `**True**`if the structure is valid.\nNext, we create a function to invoke the LLM chain and actually generate the nodes:\npython\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef get_node_definitions(chain, structure: str, example: Dict) -> Dict[str, Dict[str, str]]:\n \"\"\"Get node definitions with retry logic\"\"\"\n try:\n # Get response from LLM\n response = chain.invoke({\"structure\": structure, \"example\": example})\n\n # Parse response\n node_defs = ast.literal_eval(response)\n\n # Validate structure\n if not validate_node_definition(node_defs):\n raise ValueError(\"Invalid node definition structure\")\n\n return node_defs\n```\n\nIf you are not familiar with decorators, you may be wondering what `@retry(...)` is doing. Look at this as a wrapper function that surrounds the actual `get_node_definitions` function. In this case, we call the `retry` decorator, which automatically retries the function if an error occurs.\n * `**stop_after_attempt(3)**`: Retries up to 3 times.\n * `**wait_exponential**`: Adds an increasing delay between retries (e.g., 4s, 8s, 16s).\n\n\nThe inputs for our functions are:\n * `**chain**`: The LangChain pipeline (prompt + LLM). We will define the chain later.\n * `**structure**`: The dataset structure (columns and sample values).\n * `**example**`: A sample node definition to guide the LLM.\n\n\nNext, `**chain.invoke**`sends the`structure` and `example` to the LLM and receives a response as a string. `**ast.literal_eval**`converts the string response into a Python dictionary.\nWe check if the parsed dictionary is in the correct format using `validate_node_definition`, which raises a `ValueError` if the structure is invalid.\npython\n\n```\nexcept (ValueError, SyntaxError) as e:\n logger.error(f\"Error parsing node definitions: {e}\")\n raise\n```\n\nIf the response cannot be parsed or validated, the error is logged, and the function raises an exception.\nNext, let's provide a prompt template to the LLM to guide it on the node generation task:\npython\n\n```\ndefine_nodes_prompt = PromptTemplate(\n input_variables=[\"example\", \"structure\"],\n template=(\"\"\"\n Analyze the dataset structure below and extract the entity labels for nodes and their properties.n\n The node properties should be based on the dataset columns and their values.n\n Return the result as a dictionary where the keys are the node labels and the values are the node properties.nn\n Example: {example}nn\n Dataset Structure:n{structure}nn\n Make sure to include all the possible node labels and their properties.n\n If a property can be its own node, include it as a separate node label.n\n Please do not report triple backticks to identify a code block, just return the list of tuples.n\n Return only the dictionary containing node labels and properties, and don't include any other text or quotation.\n \"\"\"),\n)\n```\n\nNote that we are providing the node structure defined at the beginning of this section, and an example of how to generate a dictionary of nodes:\npython\n\n```\nnode_example = {\n \"NodeLabel1\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel2\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel3\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n}\n```\n\nIn the example, the keys are Node labels (e.g., `Movie`, `Director`), and the values are dictionaries of properties mapped to dataset columns (e.g., `row['property1']`).\nNext, let's execute the chain:\npython\n\n```\ntry:\n node_chain = define_nodes_prompt | llm\n node_definitions = get_node_definitions(node_chain, structure=node_structure, example=node_example)\n logger.info(f\"Node Definitions: {node_definitions}\")\nexcept Exception as e:\n logger.error(f\"Failed to get node definitions: {e}\")\n raise\n```\n\nIn LangChain we create a chain using the structure `prompt | llm | ...` , which combines the prompt template with the LLM, forming a pipeline. We use `**get_node_definitions**`to fetch and validate the node definitions.\nIf the process fails, the error is logged, and the program raises an exception.\nIf the process succeeds, it will generate something similar to this:\nbash\n\n```\nINFO:__main__:Node Definitions: {'Movie': {'Release Year': \"row['Release Year']\", 'Title': \"row['Title']\"}, 'Director': {'Name': \"row['Director']\"}, 'Cast': {'Actor': \"row['Cast']\"}, 'Genre': {'Type': \"row['Genre']\"}, 'Plot': {'Description': \"row['Plot']\"}}\n```\n\n**Generate Relationships** Once nodes are defined, we identify relationships between them. Again let's first take a look at what the whole code looks like:\npython\n\n```\nclass RelationshipIdentifier:\n \"\"\"Identifies relationships between nodes in a graph database.\"\"\"\n\n RELATIONSHIP_EXAMPLE = [\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel2\"),\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel3\"),\n (\"NodeLabel2\", \"RelationshipLabel\", \"NodeLabel3\"),\n ]\n\n PROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"example\"],\n template=\"\"\"\n Consider the following Dataset Structure:n{structure}nn\n\n Consider the following Node Definitions:n{node_definitions}nn\n\n Based on the dataset structure and node definitions, identify relationships (edges) between nodes.n\n Return the relationships as a list of triples where each triple contains the start node label, relationship label, and end node label, and each triple is a tuple.n\n Please return only the list of tuples. Please do not report triple backticks to identify a code block, just return the list of tuples.nn\n\n Example:n{example}\n \"\"\"\n)\n\n def __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n\n def validate_relationships(self, relationships: List[Tuple]) -> bool:\n \"\"\"Validate relationship structure.\"\"\"\n return all(\n isinstance(rel, tuple) and \n len(rel) == 3 and \n all(isinstance(x, str) for x in rel)\n for rel in relationships\n )\n\n @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def identify_relationships(self, structure: str, node_definitions: Dict) -> List[Tuple]:\n \"\"\"Identify relationships with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"structure\": structure, \n \"node_definitions\": str(node_definitions), \n \"example\": str(self.RELATIONSHIP_EXAMPLE)\n })\n\n relationships = ast.literal_eval(response)\n\n if not self.validate_relationships(relationships):\n raise ValueError(\"Invalid relationship structure\")\n\n self.logger.info(f\"Identified {len(relationships)} relationships\")\n return relationships\n\n except Exception as e:\n self.logger.error(f\"Error identifying relationships: {e}\")\n raise\n\n def get_relationship_types(self) -> List[str]:\n \"\"\"Extract unique relationship types.\"\"\"\n return list(set(rel[1] for rel in self.identify_relationships()))\n\n# Usage\nidentifier = RelationshipIdentifier(llm=llm)\nrelationships = identifier.identify_relationships(node_structure, node_definitions)\nprint(\"Relationships:\", relationships)\n```\n\nSince this code requires a few more operations the node generation, we organize the code in a class - `RelationshipIdentifier` - to encapsulate all the logic for relationship extraction, validation, and logging.\nWe use a similar logic, hence we provide a relationship example:\npython\n\n```\nRELATIONSHIP_EXAMPLE = [\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel2\"),\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel3\"),\n (\"NodeLabel2\", \"RelationshipLabel\", \"NodeLabel3\"),\n]\n```\n\nHere, each relationship is a tuple with:\n * **Start Node Label** : The label of the source node (e.g., `Movie`).\n * **Relationship Label** : The connection type (e.g., `DIRECTED_BY`).\n * **End Node Label** : The label of the target node (e.g., `Director`).\n\n\nNext, we define the actual prompt template:\npython\n\n```\nPROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"example\"],\n template=\"\"\"\n Consider the following Dataset Structure:n{structure}nn\n\n Consider the following Node Definitions:n{node_definitions}nn\n\n Based on the dataset structure and node definitions, identify relationships (edges) between nodes.n\n Return the relationships as a list of triples where each triple contains the start node label, relationship label, and end node label, and each triple is a tuple.n\n Please return only the list of tuples. Please do not report triple backticks to identify a code block, just return the list of tuples.nn\n\n Example:n{example}\n \"\"\"\n)\n```\n\nIn this case, we have three input variables:\n * `**structure**`: The dataset structure, listing columns and sample values. We defined it at the beginning of the section.\n * `**node_definitions**`: A dictionary of node labels and their properties. These are the nodes generated by the LLM in the previous section.\n * `**example**`: Example relationships in tuple format.\n\n\nNext, we initialize the class with three attributes:\npython\n\n```\ndef __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n```\n\n * `**llm**`: The Language Model to process the prompt (e.g., GPT-3.5 Turbo).\n * `**logger**`: Optional; logs progress and errors (defaults to a standard logger if not provided).\n * `**self.chain**`: Combines the prompt template with the LLM to create a reusable pipeline.\n\n\nSimilarly to before, we create a method to validate the generated relationships:\npython\n\n```\ndef validate_relationships(self, relationships: List[Tuple]) -> bool:\n \"\"\"Validate relationship structure.\"\"\"\n return all(\n isinstance(rel, tuple) and \n len(rel) == 3 and \n all(isinstance(x, str) for x in rel)\n for rel in relationships\n )\n```\n\nThe method checks if each item is a tuple if each tuple contains exactly three elements, and if all elements are strings (e.g., node labels or relationship types). Lastly, it returns `TRUE` if the conditions are satisfied, otherwise `FALSE` .\nNext, we create a method to invoke the chain and generate the relationships:\npython\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef identify_relationships(self, structure: str, node_definitions: Dict) -> List[Tuple]:\n \"\"\"Identify relationships with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"structure\": structure, \n \"node_definitions\": str(node_definitions), \n \"example\": str(self.RELATIONSHIP_EXAMPLE)\n })\n\n relationships = ast.literal_eval(response)\n\n if not self.validate_relationships(relationships):\n raise ValueError(\"Invalid relationship structure\")\n\n self.logger.info(f\"Identified {len(relationships)} relationships\")\n return relationships\n```\n\nWe use again the retry decorator to reattempt the LLM chain in case of failure and invoke the chain similarly to how we invoked it in the nodes generation.\nAdditionally, we use`**ast.literal_eval**`to convert the LLM's string output into a Python list and`validate_relationships` to ensure the output format is correct.\npython\n\n```\nexcept Exception as e:\n self.logger.error(f\"Error identifying relationships: {e}\")\n raise\n```\n\nIf the method fails, it logs the errors and retries the process up to 3 times.\nThe last method Returns unique relationship labels (e.g., `DIRECTED_BY`, `ACTED_IN`):\nrust\n\n```\ndef get_relationship_types(self) -> List[str]:\n \"\"\"Extract unique relationship types.\"\"\"\n return list(set(rel[1] for rel in self.identify_relationships()))\n```\n\nIt calls the`identify_relationships` method to get the list of relationships. Then, it extracts the second element (relationship label) from each tuple, uses `set` to remove duplicates, and converts the result back to a list.\nNow, it's finally time to generate the relationships:\npython\n\n```\nidentifier = RelationshipIdentifier(llm=llm)\nrelationships = identifier.identify_relationships(node_structure, node_definitions)\nprint(\"Relationships:\", relationships)\n```\n\nIf the LLM is successful within the 3 attempts it returns a list of relationships in tuple format similar to the following:\nbash\n\n```\nINFO:__main__:Identified 4 relationships\nRelationships: [('Movie', 'Directed By', 'Director'), ('Movie', 'Starring', 'Cast'), ('Movie', 'Has Genre', 'Genre'), ('Movie', 'Contains Plot', 'Plot')]\n```\n\n**Generate Cypher Queries** With nodes and relationships defined, we create Cypher queries to load them into Neo4j. The process follows a similar logic to both node generation and relationship generation. However, we define a couple more steps for validation, since the output generated will be used to load the data into our KG. Therefore, we need to maximize our chances of success. Let's first take a look at the whole code:\npython\n\n```\nclass CypherQueryBuilder:\n \"\"\"Builds Cypher queries for Neo4j graph database.\"\"\"\n\n INPUT_EXAMPLE = \"\"\"\n NodeLabel1: value1, value2\n NodeLabel2: value1, value2\n \"\"\"\n\n EXAMPLE_CYPHER = example_cypher = \"\"\"\n CREATE (n1:NodeLabel1 {property1: \"row['property1']\", property2: \"row['property2']\"})\n CREATE (n2:NodeLabel2 {property1: \"row['property1']\", property2: \"row['property2']\"})\n CREATE (n1)-[:RelationshipLabel]->(n2);\n \"\"\"\n\n PROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"relationships\", \"example\"],\n template=\"\"\"\n Consider the following Node Definitions:n{node_definitions}nn\n Consider the following Relationships:n{relationships}nn\n Generate Cypher queries to create nodes and relationships using the node definitions and relationships below. Remember to replace the placeholder values with actual data from the dataset.n\n Include all the properties in the Node Definitions for each node as defined and create relationships.n\n Return a single string with each query separated by a semicolon.n\n Don't include any other text or quotation marks in the response.n\n Please return only the string containing Cypher queries. Please do not report triple backticks to identify a code block.nn\n\n Example Input:n{input}nn\n\n Example Output Cypher query:n{cypher}\n \"\"\"\n)\n\n def __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n # self.chain = LLMChain(llm=llm, prompt=self.PROMPT_TEMPLATE)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n\n def validate_cypher_query(self, query: str) -> bool:\n \"\"\"Validate Cypher query syntax using LLM and regex patterns.\"\"\"\n\n VALIDATION_PROMPT = PromptTemplate(\n input_variables=[\"query\"],\n template=\"\"\"\n Validate this Cypher query and return TRUE or FALSE:\n\n Query: {query}\n\n Rules to check:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n Return only TRUE if query is valid, FALSE if invalid.\n \"\"\"\n )\n\n try:\n # Basic pattern validation\n basic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n ])\n\n if not basic_valid:\n return False\n\n # LLM validation\n validation_chain = VALIDATION_PROMPT | self.llm\n result = validation_chain.invoke({\"query\": query})\n\n # Parse result\n is_valid = \"TRUE\" in result.upper()\n\n if not is_valid:\n self.logger.warning(f\"LLM validation failed for query: {query}\")\n\n return is_valid\n\n except Exception as e:\n self.logger.error(f\"Validation error: {e}\")\n return False\n\n def sanitize_query(self, query: str) -> str:\n \"\"\"Sanitize and format Cypher query.\"\"\"\n return (query\n .strip()\n .replace('n', ' ')\n .replace(' ', ' ')\n .replace(\"'row[\", \"row['\")\n .replace(\"]'\", \"']\"))\n\n @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def build_queries(self, node_definitions: Dict, relationships: List) -> str:\n \"\"\"Build Cypher queries with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"node_definitions\": str(node_definitions),\n \"relationships\": str(relationships),\n \"input\": self.INPUT_EXAMPLE,\n \"cypher\": self.EXAMPLE_CYPHER\n })\n\n # Get response inside triple backticks\n if '```' in response:\n response = response.split('```')[1]\n\n # Sanitize response\n queries = self.sanitize_query(response)\n\n # Validate queries\n if not self.validate_cypher_query(queries):\n raise ValueError(\"Invalid Cypher query syntax\")\n\n self.logger.info(\"Successfully generated Cypher queries\")\n return queries\n\n except Exception as e:\n self.logger.error(f\"Error building Cypher queries: {e}\")\n raise\n\n def split_queries(self, queries: str) -> List[str]:\n \"\"\"Split combined queries into individual statements.\"\"\"\n return [q.strip() for q in queries.split(';') if q.strip()]\n\n# Usage\nbuilder = CypherQueryBuilder(llm=llm)\ncypher_queries = builder.build_queries(node_definitions, relationships)\nprint(\"Cypher Queries:\", cypher_queries)\n```\n\nWe provide a prompt template to help the LLM:\npython\n\n```\nPROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"relationships\", \"example\"],\n template=\"\"\"\n Consider the following Node Definitions:n{node_definitions}nn\n Consider the following Relationships:n{relationships}nn\n Generate Cypher queries to create nodes and relationships using the node definitions and relationships below. Remember to replace the placeholder values with actual data from the dataset.n\n Include all the properties in the Node Definitions for each node as defined and create relationships.n\n Return a single string with each query separated by a semicolon.n\n Don't include any other text or quotation marks in the response.n\n Please return only the string containing Cypher queries. Please do not report triple backticks to identify a code block.nn\n\n Example Input:n{input}nn\n\n Example Output Cypher query:n{cypher}\n \"\"\"\n)\n```\n\nNow we provide four inputs to the prompt:\n * `**structure**`: Dataset structure for context.\n * `**node_definitions**`: Generated Nodes and their properties.\n * `**relationships**`: Generated Edges between nodes.\n * `**example**`: Example queries for formatting reference.\n\n\npython\n\n```\ndef __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n```\n\nWe initialize the class in the same fashion as the relationship class.\nNext, we define a validation method to check the generated output:\npython\n\n```\ndef validate_cypher_query(self, query: str) -> bool:\n \"\"\"Validate Cypher query syntax using LLM and regex patterns.\"\"\"\n\n VALIDATION_PROMPT = PromptTemplate(\n input_variables=[\"query\"],\n template=\"\"\"\n Validate this Cypher query and return TRUE or FALSE:\n\n Query: {query}\n\n Rules to check:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n Return only TRUE if query is valid, FALSE if invalid.\n \"\"\"56\n )\n\n try:\n # Basic pattern validation\n basic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n ])\n\n if not basic_valid:\n return False\n\n # LLM validation\n validation_chain = VALIDATION_PROMPT | self.llm\n result = validation_chain.invoke({\"query\": query})\n\n # Parse result\n is_valid = \"TRUE\" in result.upper()\n\n if not is_valid:\n self.logger.warning(f\"LLM validation failed for query: {query}\")\n\n return is_valid\n\n except Exception as e:\n self.logger.error(f\"Validation error: {e}\")\n return False\n```\n\nThis method does two validation steps. First a basic validation with regular expressions:\npython\n\n```\nbasic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n])\nif not basic_valid:\n return False\n```\n\nThis ensures the query contains essential Cypher syntax:\n * `**CREATE**`: Ensures nodes and relationships are being created.\n * *_`{._?}`**: Ensures properties are included.\n * *_`-[:._?]->`**: Ensures relationships are correctly formatted.\n\n\nThen, it performs an advanced validation with LLM:\npython\n\n```\nvalidation_chain = VALIDATION_PROMPT | self.llm\nresult = validation_chain.invoke({\"query\": query})\nis_valid = \"TRUE\" in result.upper()\n```\n\nThe validation is specified in the prompt where we ask the LLM to make sure that we have:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n\nEven though we should be in a good state as of now, let's add a method, to further sanitize the generated output:\nscss\n\n```\ndef sanitize_query(self, query: str) -> str:\n \"\"\"Sanitize and format Cypher query.\"\"\"\n return (query\n .strip()\n .replace('n', ' ')\n .replace(' ', ' ')\n .replace(\"'row[\", \"row['\")\n .replace(\"]'\", \"']\"))\n```\n\nIn particular, we are removing unnecessary spaces, as well as line breaks (`n`), and fix potential formatting issues with dataset references (e.g., `row['property1']`).\nPlease consider updating this method based on the model you are using. Smaller models will likely require more sanitization.\nNext, we define a query invocation method:\nless\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def build_queries(self, node_definitions: Dict, relationships: List) -> str:\n \"\"\"Build Cypher queries with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"node_definitions\": str(node_definitions),\n \"relationships\": str(relationships),\n \"input\": self.INPUT_EXAMPLE,\n \"cypher\": self.EXAMPLE_CYPHER\n })\n\n # Get response inside triple backticks\n if '```' in response:\n response = response.split('```')[1]\n\n # Sanitize response\n queries = self.sanitize_query(response)\n\n # Validate queries\n if not self.validate_cypher_query(queries):\n raise ValueError(\"Invalid Cypher query syntax\")\n\n self.logger.info(\"Successfully generated Cypher queries\")\n return queries\n\n except Exception as e:\n self.logger.error(f\"Error building Cypher queries: {e}\")\n raise\n```\n\nThis method works similarly to the one in the relationship builder class, with the only addition of:\npython\n\n```\nif '```' in response:\n response = response.split('```')[1]\n```\n\nHere, the LLM may provide an additional markdown format to specify it's a code block. If this is present in the LLM's response, we only retrieve the code within the triple backticks.\nNext, we define a method to break a single string of Cypher queries into individual statements:\npython\n\n```\ndef split_queries(self, queries: str) -> List[str]:\n \"\"\"Split combined queries into individual statements.\"\"\"\n return [q.strip() for q in queries.split(';') if q.strip()]\n```\n\nFor example, this Cypher query:\nsql\n\n```\nCREATE (n1:Movie {title: \"Inception\"}); CREATE (n2:Director {name: \"Nolan\"});\n```\n\nThis will turn into this:\nbash\n\n```\n[\"CREATE (n1:Movie {title: 'Inception'})\", \"CREATE (n2:Director {name: 'Nolan'})\"]\n```\n\nThis will be useful so we can iterate over a list of queries.\nLastly, we initialize the class and generate the Cypher queries:\npython\n\n```\nbuilder = CypherQueryBuilder(llm=llm)\ncypher_queries = builder.build_queries(node_definitions, relationships)\nprint(\"Cypher Queries:\", cypher_queries)\n```\n\nIn case of success, the output will look like this:\nbash\n\n```\nINFO:__main__:Successfully generated Cypher queries\nCypher Queries: CREATE (m:Movie {Release_Year: \"row['Release Year']\", Title: \"row['Title']\"}) CREATE (d:Director {Name: \"row['Director']\"}) CREATE (c:Cast {Actor: \"row['Cast']\"}) CREATE (g:Genre {Type: \"row['Genre']\"}) CREATE (p:Plot {Description: \"row['Plot']\"}) CREATE (m)-[:Directed_By]->(d) CREATE (m)-[:Starring]->(c) CREATE (m)-[:Has_Genre]->(g) CREATE (m)-[:Contains_Plot]->(p)\n```\n\nFinally, we iterate over the dataset and execute the generated Cypher queries for each row.\npython\n\n```\nlogs = \"\"\ntotal_rows = len(df)\n\ndef sanitize_value(value):\n if isinstance(value, str):\n return value.replace('\"', '')\n return str(value)\n\nfor index, row in tqdm(df.iterrows(), \n total=total_rows,\n desc=\"Loading data to Neo4j\",\n position=0,\n leave=True):\n\n # Replace placeholders with actual values\n cypher_query = cypher_queries\n for column in df.columns:\n cypher_query = cypher_query.replace(\n f\"row['{column}']\", \n f'{sanitize_value(row[column])}'\n )\n\n try:\n # Execute query and update progress\n conn.execute_query(cypher_query)\n except Exception as e:\n logs += f\"Error on row {index+1}: {str(e)}n\"\n```\n\nNote that we define an empty string variable `logs` which we will use to capture potential failures. Also, we add a sanitize function to pass to each value in the row input:\npython\n\n```\ndef sanitize_value(value):\n if isinstance(value, str):\n return value.replace('\"', '')\n return str(value)\n```\n\nThis will prevent strings containing double quotes from breaking the query syntax.\nNext, we loop over the dataset:\npython\n\n```\nfor index, row in tqdm(df.iterrows(), \n total=total_rows,\n desc=\"Loading data to Neo4j\",\n position=0,\n leave=True):\n\n # Replace placeholders with actual values\n cypher_query = cypher_queries\n for column in df.columns:\n cypher_query = cypher_query.replace(\n f\"row['{column}']\", \n f'{sanitize_value(row[column])}'\n )\n\n try:\n # Execute query and update progress\n conn.execute_query(cypher_query)\n except Exception as e:\n logs += f\"Error on row {index+1}: {str(e)}n\"\n```\n\nAs I mentioned at the beginning of the exercise, we use `tqdm` to add a nice-looking progress bar to visualize how many rows have been processed. We pass `**df.iterrows()**`to iterate through the DataFrame, providing the index and the row data.`**total=total_rows**`is used by`tqdm` to calculate progress. We add`**desc=\"Loading data to Neo4j\"**`to provide a label for the progress bar. Lastly,`**position=0, leave=True**`ensures the progress bar stays visible in the console.\nNext, we replace placeholders like `row['column_name']` with the actual dataset values passing each value in the `sanitize_value` function, and execute the query.\nLet's check if our dataset is now uploaded. Switch to the Neo4j browser, and run the following Cypher query:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nIn my case, the LLM generated the following graph:\n![Custom LLM generated Knowledge Graph - Image by Author](../images/003.jpeg)\nCustom LLM generated Knowledge Graph - Image by Author\nThis is pretty similar to the knowledge graph we uploaded manually. Not bad for a Naïve LLM right? Even though that required quite some code, we can now reuse it for multiple datasets, and more importantly, we can use it as a base to create more complex LLM graph builders.\nIn our example, we haven't helped our LLM by providing entities, relationships, and properties for both. However, consider adding them as examples to increase the performance of the LLM. Moreover, modern approaches leverage the chain of thought to come up with additional nodes and relationships, this makes the model sequentially reason over the output to further improve it. Another strategy can be providing samples of rows to better adapt to the values provided in each row.\nIn the next example, we will see a modern implementation of Graph-RAG with LangChain.\n### LLMGraphTransformer with LangChain\nTo make our graph smarter, we use LangChain to extract entities (movies, actors, etc.) and relationships from text descriptions. `LLMGraphTransformer` is designed to **transform text-based documents into graph documents** using a Language Model.\nLet's start by initializing it:\npython\n\n```\nllm_transformer = LLMGraphTransformer(\n llm=llm,\n)\n```\n\nThe LLM provided is the same one we used for our custom Graph Builder. In this case, we are using the default parameters to give freedom to the model to experiment with nodes, edges, and properties. However, there are a few parameters that are worth knowing to potentially further boost the performances of this algorithm:\n * `**allowed_nodes**`and`**allowed_relationships**`: We haven't specified, so by default, all node and relationship types are allowed.\n * `**strict_mode=True**`: Ensures that only allowed nodes and relationships are included in the output if constraints are specified.\n * `**node_properties=False**`: Disables property extraction for nodes.\n * `**relationship_properties=False**`: Disables property extraction for relationships.\n * `**prompt**`: Pass a`ChatPromptTemplate` to customize the context of the LLM. This is similar to what we have done in our custom LLM.\n\n\nOne caveat of this algorithm is that it's quite slow, especially considering we are not providing a list of nodes and relationships. Therefore, we will only use 100 rows out of the 1000 available in the dataset to speed things up:\npython\n\n```\ndf_sample = df.head(100) # Reduce sample size for faster processing\n```\n\nNext, let's prepare our dataset. We said \"`LLMGraphTransformer` is designed to **transform text-based documents into graph documents\"** , this means that we need to turn our pandas dataframe into text:\npython\n\n```\ndf_sample = movies.head(100)\n\ndocuments = []\nfor _, row in tqdm(df_sample.iterrows(), \n total=len(df_sample), \n desc=\"Creating documents\",\n position=0, \n leave=True):\n try:\n # Format text with proper line breaks\n text = \"\"\n\n for col in df.columns:\n text += f\"{col}: {row[col]}n\"\n\n documents.append(Document(page_content=text))\n\n except KeyError as e:\n tqdm.write(f\"Missing column: {e}\")\n except Exception as e:\n tqdm.write(f\"Error processing row: {e}\")\n```\n\nThis will convert each row into text and add it to a LangChain `Document` object, which is compatible with LangChain's `LLMGraphTransformer` .\nNext, we run the LLM and start the generation:\npython\n\n```\ngraph_documents = await llm_transformer.aconvert_to_graph_documents(documents)\n```\n\nNote that in this case, we are using `await`, and `aconvert_to_graph_documents` instead of `convert_to_graph_documents` to process documents asynchronously, enabling faster execution in large-scale applications.\nNext, sit tight because this will take a few minutes (~30 minutes). Once, the conversion is finished let's print the generated nodes and relationships:\npython\n\n```\nprint(f\"Nodes:{graph_documents[0].nodes}\")\nprint(f\"Relationships:{graph_documents[0].relationships}\")\n```\n\nIn my case, I got the following:\nbash\n\n```\nNodes:[Node(id=\"Boone's cabin\", type='Cabin', properties={}), Node(id='Boone', type='Boone', properties={}), Node(id='Indian maiden', type='Person', properties={}), Node(id='Indian chief', type='Chief', properties={}), Node(id='Florence Lawrence', type='Person', properties={}), Node(id='William Craven', type='Person', properties={}), Node(id='Wallace mccutcheon and ediwin s. porter', type='Director', properties={}), Node(id='Burning arrow', type='Arrow', properties={}), Node(id='Boone', type='Person', properties={}), Node(id='Indian camp', type='Camp', properties={}), Node(id='American', type='Ethnicity', properties={}), Node(id=\"Boone's horse\", type='Horse', properties={}), Node(id='None', type='None', properties={}), Node(id='an indian maiden', type='Maiden', properties={}), Node(id='William craven', type='Cast', properties={}), Node(id='florence lawrence', type='Cast', properties={}), Node(id='Swears vengeance', type='Vengeance', properties={}), Node(id='Daniel Boone', type='Person', properties={}), Node(id='Indians', type='Group', properties={}), Node(id='Daniel boone', type='Title', properties={}), Node(id=\"Daniel Boone's daughter\", type='Person', properties={})]\nRelationships:[Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='Daniel boone', type='Title', properties={}), type='TITLE', properties={}), Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='American', type='Ethnicity', properties={}), type='ORIGIN_ETHNICITY', properties={}), Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='Wallace mccutcheon and ediwin s. porter', type='Director', properties={}), type='DIRECTED_BY', properties={}), Relationship(source=Node(id='William Craven', type='Person', properties={}), target=Node(id='William craven', type='Cast', properties={}), type='CAST', properties={}), Relationship(source=Node(id='Florence Lawrence', type='Person', properties={}), target=Node(id='florence lawrence', type='Cast', properties={}), type='CAST', properties={}), Relationship(source=Node(id=\"Daniel Boone's daughter\", type='Person', properties={}), target=Node(id='an indian maiden', type='Maiden', properties={}), type='BEFRIENDS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Indian camp', type='Camp', properties={}), type='LEADS_OUT_ON_A_HUNTING_EXPEDITION', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id=\"Boone's cabin\", type='Cabin', properties={}), type='ATTACKS', properties={}), Relationship(source=Node(id='Indian maiden', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='ESCAPES', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Swears vengeance', type='Vengeance', properties={}), type='RETURNS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='HEADS_OUT_ON_THE_TRAIL_TO_THE_INDIAN_CAMP', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id='Boone', type='Boone', properties={}), type='ENCOUNTERS', properties={}), Relationship(source=Node(id='Indian camp', type='Camp', properties={}), target=Node(id='Burning arrow', type='Arrow', properties={}), type='SET_ON_FIRE', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='GETS_TIED_TO_THE_STAKE_AND_TOURED', properties={}), Relationship(source=Node(id='Indian camp', type='Camp', properties={}), target=Node(id='Burning arrow', type='Arrow', properties={}), type='SETS_ON_FIRE', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id=\"Boone's horse\", type='Horse', properties={}), type='ENCOUNTERS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Indian chief', type='Chief', properties={}), type='HAS_KNIFE_FIGHT_IN_WHICH_HE_KILLS_THE_INDIAN_CHIEF', properties={})]\n```\n\nNext, it's time to add the generated graph documents to our knowledge graph. We can do that by leveraging the LangChain integration of Neo4j:\npython\n\n```\ngraph = Neo4jGraph(url=uri, username=user, password=password, enhanced_schema=True)\ngraph.add_graph_documents(graph_documents)\n```\n\nLet's store the graph connection in the `graph` variable passing the same URL, username, and password you used at the beginning of this application. Then, let's call the `add_graph_documents` method to add all the graph documents to our database.\nOnce, the process is complete, let's switch to Neo4j Browser one last time, and check our new knowledge graph.\nAs always, run the following query to see the knowledge graph:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nIn my case, the KG looks like this:\n![Knowledge Graph generated by `LLMGraphTransformer - Image by Author`](../images/004.jpeg)\nKnowledge Graph generated by **`LLMGraphTransformer** - Image by Author`\nWell, that's a knowledge graph.\nBut we are not done yet. You may remember that our mission is to actually interrogate the knowledge graph to help us find movies. In this article, I will provide a simple Text to Cypher approach which will leverage an LLM to generate a Cypher query, run the Cypher query, and use the retrieved information as context to answer the user query. However, consider that this is just a simple approach, and we will explore advanced retrieval methods in a future article.\nFirst of all, let's refresh the schema of the graph since we will use it to give an understanding of our KG to the LLM:\npython\n\n```\ngraph.refresh_schema()\n```\n\nNow, it's time to set up the QA chain:\npython\n\n```\n# llm = ChatGoogleGenerativeAI(\n# model=\"gemini-1.5-pro\",\n# temperature=0,\n# max_tokens=None,\n# timeout=None,\n# max_retries=2,\n# api_key=api_key\n# )\n\nCYPHER_GENERATION_TEMPLATE = \"\"\"Task:Generate Cypher statement to query a graph database.\nInstructions:\nUse only the provided relationship types and properties in the schema.\nDo not use any other relationship types or properties that are not provided.\nSchema:\n{schema}\nNote: Do not include any explanations or apologies in your responses.\nDo not respond to any questions that might ask anything else than for you to construct a Cypher statement.\nDo not include any text except the generated Cypher statement.\nReturn every node as whole, do not return only the properties.\n\nThe question is:\n{question}\"\"\"\n\nCYPHER_GENERATION_PROMPT = PromptTemplate(\n input_variables=[\"schema\", \"question\"], template=CYPHER_GENERATION_TEMPLATE\n)\n\nchain = GraphCypherQAChain.from_llm(\n llm, \n graph=graph, \n verbose=True, \n allow_dangerous_requests=True, \n return_intermediate_steps=True,\n cypher_prompt=CYPHER_GENERATION_PROMPT\n)\n```\n\nIf you are using Gemini, make sure to switch to a chat model by uncommenting the `llm` variable at the top.\nIn the code above we provide a prompt to the LLM to help it with the generation, and the schema of the graph passing the graph variable.\nFinally, let's ask it something:\npython\n\n```\nchain.run(\"Give me an overview of the movie titled David Copperfield.\")\n```\n\nIn my case, the output is:\nbash\n\n```\nGenerated Cypher:\nMATCH p=(n:Title {id: 'David Copperfield'})-[*1..2]-()\nRETURN p\nFull Context:\n[{'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Florence La Badie'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'RELEASE_YEAR', {'id': '1911'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'GENRE', {'id': 'Drama'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'DIRECTED', {'id': 'Theodore Marston'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'ORIGIN_ETHNICITY', {'id': 'American'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Mignon Anderson'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'William Russell'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Marie Eline'}]}]\nINFO:httpx:HTTP Request: POST http://127.0.0.1:11434/api/generate \"HTTP/1.1 200 OK\"\n\n> Finished chain.\n'David Copperfield is a 1911 American drama film directed by Theodore Marston. The movie stars David Copperfield, Florence La Badie, Mignon Anderson, William Russell, and Marie Eline in various roles. It provides an overview of the life and adventures of David Copperfield through the eyes of his various companions and experiences.'\n```\n\nBy setting `verbose=True` we output the intermediate step: generated Cypher query, and context provided.\n## Conclusion\nThis article served as an introduction to a modern approach to building a knowledge graph. First, we explored a traditional approach and had an overview of the Cypher language, then created a Naïve LLM graph builder to automate the Graph-building process matching the performances achieved during the manual process. Lastly, we went a step further and introduced `LLMGraphTransformer` by LangChain which significantly improved our KG. However, this is just the beginning of our Graph RAG journey specifically our graph builder journey. There are many more modern approaches we need to explore and build from scratch, and we will do that in future articles.\nMoreover, the strategies we talked about above are still not considering the second component of Graph-RAG: the graph retriever. Although improving the actual graph will also improve retrieval performances, we are still not thinking of a retrieval point of view. For example, following the direction we have taken for now, the more labels, nodes, edges, and properties, the better. However, that's actually making it harder for the retriever function to identify the right information to retrieve. Indeed, modern approaches also consider following a tree structure in the knowledge graph, by creating macro-areas and further breaking them down into micro-areas to help the retrieval.\nAs of now, I would like you to test all this code on a different dataset, and get creative with the customization of our Naïve LLM graph-builder. Although `LLMGraphTransformer` seems a very convenient choice to speed up a flexible graph-builder with a minimum amount of code, this method required much more time to build our KG. Moreover, building it from scratch will make sure you are grasping and fully understanding every component behind Graph-RAG.\nGet ready for more Graph-RAG in future articles!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-8f73ac5a40c91a7b503f.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Llm](https://towardsdatascience.com/tag/llm/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Retrieval Augmented Gen](https://towardsdatascience.com/tag/retrieval-augmented-gen/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&title=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&title=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&text=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/005.jpeg)](https://towardsdatascience.com/bias-detection-in-machine-learning-models-using-amazon-sagemaker-clarify-d96482692611/)\n## [Bias Detection in Machine Learning Models using Amazon SageMaker Clarify](https://towardsdatascience.com/bias-detection-in-machine-learning-models-using-amazon-sagemaker-clarify-d96482692611/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n1. Introduction\n[Gaurav Shekhar](https://towardsdatascience.com/author/gaurav.shekhar/)\nMarch 11, 20219 min read\n * [![](../images/006.jpg)](https://towardsdatascience.com/introduction-to-dropout-to-regularize-deep-neural-network-8e9d6b1d4386/)\n## [Why Dropout is so effective in Deep Neural Network?](https://towardsdatascience.com/introduction-to-dropout-to-regularize-deep-neural-network-8e9d6b1d4386/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDropout is a simple way to reduce dependencies in the Neural Network\n[Saurav Singla](https://towardsdatascience.com/author/sauravsingla08/)\nAugust 2, 20205 min read\n * [![](../images/007.png)](https://towardsdatascience.com/microsofts-new-ai-method-to-predict-how-molecules-move-and-function-93d47e246b5d/)\n## [Microsoft's New AI Method to Predict How Molecules Move and Function](https://towardsdatascience.com/microsofts-new-ai-method-to-predict-how-molecules-move-and-function-93d47e246b5d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTermed the \"Distributional Graphormer\", it could kick-start the next step in the revolution that AI is bringing to the chemical and...\n[Luciano Abriata](https://towardsdatascience.com/author/lucianosphere/)\nJuly 20, 20239 min read\n * [![](../images/008.png)](https://towardsdatascience.com/ai-computed-breast-cancer-risk-map-b29195b477a/)\n## [AI Computed Breast Cancer Risk Map](https://towardsdatascience.com/ai-computed-breast-cancer-risk-map-b29195b477a/)\n[](https://towardsdatascience.com/building-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07/)\nA visual and reliable breast cancer diagnosis model using Hierarchical Clustering combined with Boosting and Support Vector Machines\n[jean-baptiste charraud](https://towardsdatascience.com/author/jbcharraud/)\nApril 10, 202118 min read\n * [![](../images/009.jpg)](https://towardsdatascience.com/how-machine-learning-will-become-a-game-changer-for-the-location-data-industry-10dde8f41f3e/)\n## [How Machine Learning Will Become a Game Changer for the Location Data Industry](https://towardsdatascience.com/how-machine-learning-will-become-a-game-changer-for-the-location-data-industry-10dde8f41f3e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLocation data can provide unique insights but comes with costs and privacy issues. ML can overcome these drawbacks and improve location...\n[Benjamin Thürer](https://towardsdatascience.com/author/benjamin.thuerer/)\nMarch 9, 202310 min read\n * [![](../images/010.png)](https://towardsdatascience.com/explaining-openai-soras-spacetime-patches-the-key-ingredient-e14e0703ec5b/)\n## [Explaining OpenAI Sora's Spacetime Patches: The Key Ingredient](https://towardsdatascience.com/explaining-openai-soras-spacetime-patches-the-key-ingredient-e14e0703ec5b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnder The Hood Of The Generative AI For Video By OpenAI\n[Vincent Koc](https://towardsdatascience.com/author/vincentkoc/)\nFebruary 16, 20246 min read\n * [![](../images/011.jpeg)](https://towardsdatascience.com/how-to-become-a-pandas-jedi-bad82667b34c/)\n## [How to Become a Pandas Jedi](https://towardsdatascience.com/how-to-become-a-pandas-jedi-bad82667b34c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA practical guide to boost your data analysis skills\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nFebruary 7, 20215 min read\n * [![](../images/012.jpeg)](https://towardsdatascience.com/stop-using-csvs-for-storage-this-file-format-is-150-times-faster-158bd322074e/)\n## [Stop Using CSVs for Storage - This File Format Is 150 Times Faster](https://towardsdatascience.com/stop-using-csvs-for-storage-this-file-format-is-150-times-faster-158bd322074e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nCSV's are costing you time, disk space, and money. It's time to end it.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nSeptember 6, 20214 min read\n * [![](../images/013.png)](https://towardsdatascience.com/intuitively-understand-roc-and-implement-it-in-r-and-python-b6aa1e8508d/)\n## [Intuitively understand ROC and implement it in R and Python](https://towardsdatascience.com/intuitively-understand-roc-and-implement-it-in-r-and-python-b6aa1e8508d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA case-study based introduction to ROC, how to set the threshold, its relation to sensitivity/specificity/precision/recall/accuracy, and...\n[Ahmar Shah, PhD (Oxford)](https://towardsdatascience.com/author/ahmarshah/)\nDecember 9, 202012 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "Building a Knowledge Graph From Scratch Using LLMs | Towards Data Science", "description": "Turn your Pandas Data Frame into a Knowledge Graph using LLMs. Build your own LLM graph-builder and QA your KG.", "keywords": null, "author": "Cristian Leo", "og:locale": "en_US", "og:type": "article", "og:title": "Building a Knowledge Graph From Scratch Using LLMs", "og:description": "Turn your Pandas Data Frame into a Knowledge Graph using LLMs. Build your own LLM graph-builder and QA your KG.", "og:url": "https://towardsdatascience.com/building-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/169AP1WpV7NfZZi8ptYeiGA.png", "og:image:width": "1553", "og:image:height": "919", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Cristian Leo", "twitter:label2": "Est. reading time", "twitter:data2": "39 min read", "article:published_time": "2024-11-25T19:13:53.000Z", "article:modified_time": "2024-11-25T19:13:53.000Z", "article:author": "https://towardsdatascience.com/author/cristianleo120/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/0852990cc19f/code/block_000.txt", "chars": 12 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_001.txt", "chars": 32 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_002.txt", "chars": 12 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_003.txt", "chars": 21 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_004.txt", "chars": 1036 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_005.txt", "chars": 116 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_006.txt", "chars": 844 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_007.txt", "chars": 616 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_008.txt", "chars": 143 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_009.txt", "chars": 482 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_010.txt", "chars": 2281 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_011.txt", "chars": 35 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_012.txt", "chars": 164 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_013.txt", "chars": 647 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_014.txt", "chars": 46 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_015.txt", "chars": 153 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_016.txt", "chars": 149 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_017.txt", "chars": 2086 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_018.txt", "chars": 2639 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_019.txt", "chars": 77 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_020.txt", "chars": 290 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_021.txt", "chars": 594 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_022.txt", "chars": 106 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_023.txt", "chars": 884 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_024.txt", "chars": 277 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_025.txt", "chars": 299 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_026.txt", "chars": 258 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_027.txt", "chars": 2792 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_028.txt", "chars": 192 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_029.txt", "chars": 720 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_030.txt", "chars": 185 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_031.txt", "chars": 276 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_032.txt", "chars": 713 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_033.txt", "chars": 96 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_034.txt", "chars": 162 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_035.txt", "chars": 168 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_036.txt", "chars": 195 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_037.txt", "chars": 4009 }, { "language": "')[1]", "file": "attachments/0852990cc19f/code/block_038.txt", "chars": 804 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_039.txt", "chars": 946 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_040.txt", "chars": 185 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_041.txt", "chars": 1320 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_042.txt", "chars": 156 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_043.txt", "chars": 135 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_044.txt", "chars": 264 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_045.txt", "chars": 563 }, { "language": "')[1]", "file": "attachments/0852990cc19f/code/block_046.txt", "chars": 454 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_047.txt", "chars": 4 }, { "language": "')[1]", "file": "attachments/0852990cc19f/code/block_048.txt", "chars": 0 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_049.txt", "chars": 177 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_050.txt", "chars": 78 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_051.txt", "chars": 83 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_052.txt", "chars": 151 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_053.txt", "chars": 437 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_054.txt", "chars": 770 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_055.txt", "chars": 118 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_056.txt", "chars": 619 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_057.txt", "chars": 46 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_058.txt", "chars": 54 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_059.txt", "chars": 69 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_060.txt", "chars": 587 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_061.txt", "chars": 79 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_062.txt", "chars": 102 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_063.txt", "chars": 4185 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_064.txt", "chars": 127 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_065.txt", "chars": 46 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_066.txt", "chars": 23 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_067.txt", "chars": 1106 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_068.txt", "chars": 72 }, { "language": "txt", "file": "attachments/0852990cc19f/code/block_069.txt", "chars": 1506 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:52:53Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/cristianleo120/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/hands-on-tutorials/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/llm/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/retrieval-augmented-gen/" }, { "type": "internal", "href": "https://towardsdatascience.com/bias-detection-in-machine-learning-models-using-amazon-sagemaker-clarify-d96482692611/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/gaurav.shekhar/" }, { "type": "internal", "href": "https://towardsdatascience.com/introduction-to-dropout-to-regularize-deep-neural-network-8e9d6b1d4386/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/sauravsingla08/" }, { "type": "internal", "href": "https://towardsdatascience.com/microsofts-new-ai-method-to-predict-how-molecules-move-and-function-93d47e246b5d/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/lucianosphere/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://roadmap.sh/ai-engineer" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots" }, { "type": "external", "href": "https://creativecommons.org/licenses/by-sa/4.0/" }, { "type": "external", "href": "https://github.com/cristianleoo/rag-knowledge-graph" }, { "type": "external", "href": "https://neo4j.com/download/?gad_source=1&utm_adgroup=DSA" }, { "type": "external", "href": "https://ollama.ai/" }, { "type": "external", "href": "https://www.googleadservices.com/pagead/aclk?adurl=&ai=DChcSEwiL4OC_neSJAxV-cEcBHSPmEjUYABAAGgJxdQ&ase=2&cid=CAESVuD2RgfXSVKvkdgIqHdVxFkchLNjCfya9TRqTM6SsW3mXjsq4CFoeiUD5rFUM47GMTu1RN9W0JE9DzrlzBDI75iT7oDTASAPGQU8SpT5_Z6M101LwjzM&co=1&ei=SlU6Z8KoOdWt5NoP0oio6QY&nis=4&ohost=www.google.com&q=&sa=L&sig=AOD64_3PJYSJ7hFfGD_flW0QfFen0Mypnw&sqi=2&ved=2ahUKEwjC9dq_neSJAxXVFlkFHVIEKm0Q0Qx6BAgLEAE" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Building+a+Knowledge+Graph+From+Scratch+Using+LLMs&u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=Building+a+Knowledge+Graph+From+Scratch+Using+LLMs&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F" }, { "type": "external", "href": "https://x.com/share?text=Building+a+Knowledge+Graph+From+Scratch+Using+LLMs&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" }, { "type": "external", "href": "https://insightmediagroup.io/creator-network" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:27:48Z", "source_url": "https://towardsdatascience.com/building-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Building a Knowledge Graph From Scratch Using LLMs\nTurn your Pandas Data Frame into a Knowledge Graph using LLMs. Build your own LLM graph-builder and QA your KG.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nNovember 25, 202439 min read\n### Turn your Pandas data frame into a knowledge graph using LLMs. Build your own LLM graph-builder from scratch, implement LLMGraphTransformer by LangChain, and QA your KG.\n![Knowledge Graph about 1000 Movies from Wikipedia - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/169AP1WpV7NfZZi8ptYeiGA.png)\nKnowledge Graph about 1000 Movies from Wikipedia - Image by Author\nIn today's AI world, knowledge graphs are becoming increasingly important, as they enable many of the knowledge retrieval systems behind LLMs. Many data science teams across companies are investing heavily in retrieval augmented generation (RAG), as it's an efficient way to improve the output accuracy of LLMs and prevent hallucinations.\nBut there is more to it; on a personal note, graph-RAG is democratizing the AI space. This is because before if we wanted to customize a model to a use-case - either for fun or business - we would have three options: pre-training the model providing a bigger exposure to a dataset within the industry of your use-case, fine-tuning the model on a specific dataset, and context prompting.\nAs for pre-training, this option is incredibly expensive and technical, and it's not an option for most developers.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\nFine-tuning is easier than pre-training, and although the cost of fine-tuning depends on the model and the training corpus, it's generally a more affordable option. This one was the go-to strategy for most of the AI developers. However, new models are released every week and you would need to fine-tune a new model constantly.\nThe third option involves providing the knowledge directly in the prompt. However, this works only when the knowledge required for the model to learn is relatively small. Even though the context for the models is getting bigger and bigger, the accuracy of recalling an element is inversely correlated to the size of the context provided.\nNeither of the three options sounds like the right one. Is there another option for the model to learn all the knowledge required to be specialized in a certain task or topic? No.\nBut, the model doesn't need to learn all the knowledge at once, as when we interrogate the LLM we are likely trying to get one or a few pieces of information. Here comes graph-RAG to help, by providing an informational retrieval that based on the query retrieves the information needed, without requiring any further training.\nLet's take a look at what a **Graph-RAG** looks like:\n * **Graph Building** : here, we create nodes (entities) and edges (relationships) from a data source and load them into our KG. This is usually a more manual step where we use a query language, often OpenCypher to upload both entities and connect them to each other via edges.\n * **Node indexing** : this step involves creating a data structure that will allow us to retrieve the data efficiently. In knowledge graphs, this typically involves creating a vector search index, where every index is associated with a vector embedding.\n * **Graph retriever** : here, we build a retrieval function that will allow us to compute a similarity score and retrieve the most relevant nodes that will be used as context for the LLM to provide an answer. In the simplest case, we compute the cosine similarity over the query, which is converted to a vector embedding, and all the vector embeddings in the vector index.\n * **RAG Evaluation** : the last step is useful to actually measure the accuracy and performance of the LLM. It's useful during experimentation to perform how different LLMs, and RAG frameworks perform on your specific use case.\n\n\nNow that we have an overall view of what a RAG pipeline consists of, we may be attracted to just jump and experiment with complex math functions for graph retrieval to guarantee the best information retrieval accuracy possible. But ... hold on. We don't have a knowledge graph yet. This step may seem like the classic data cleaning and preprocessing step in data science (boring...). But what if I tell you there is a better alternative? An option that introduces more science and automation. Indeed, recent studies are focusing on how to automate the construction of a Knowledge Graph, since this step is key for good information retrieval. Just think about it, if the data in the KG is not good, there is no way your graph-RAG will have a state-of-the-art performance.\nIn this article, we will delve into the first step: How to build a knowledge graph without actually building it.\n## From CSV to Knowledge Graph\nNow, let's go over a practical example to make things more concrete. Let's tackle one of the most important existential issues: what movie to watch? ... How many times have you been bored, and tired from work and the only thing you could do was watch a movie? You start scrolling among movies until you realize that two hours have passed by.\nTo solve this issue, let's create a knowledge graph using a [Wikipedia Movie Dataset](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots), and chat with the KG. Firstly, let's implement a \"from scratch\" solution using LLMs. Then, let's look at one of the latest implementations via LangChain (still in the experimental phase as of November 2024), one of the most popular and powerful LLM frameworks available, and another popular solution with LlamaIndex.\nLet's download this public dataset from Kaggle ([License: CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)):\n> [**Wikipedia Movie Plots**](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots)\nOr if you are lazy, just go ahead and clone my GitHub repo:\n> [**GitHub - cristianleoo/rag-knowledge-graph**](https://github.com/cristianleoo/rag-knowledge-graph)\nThe folder `knowledge-builder` contains both the Jupyter Notebook and the data we will be covering in this article.\n### Prerequisites\nBefore we can start, we need to have access to Neo4j Desktop and an LLM API Key or a local LLM. If you already have them, feel free to skip this section and jump to the action. If not, let's set them up, and don't worry it will be completely free.\nThere are several ways to leverage Neo4j, but for the sake of simplicity we will use Neo4j desktop, hence we will host the database locally. But it's a small dataset, so you won't be destroying your laptop by running this application.\nTo install Neo4j simply visit the [Neo4j Desktop Download Page](https://neo4j.com/download/?utm_source=GSearch&utm_medium=PaidSearch&utm_campaign=Evergreen&utm_content=AMS-Search-SEMCE-DSA-None-SEM-SEM-NonABM&utm_term=&utm_adgroup=DSA&gad_source=1&gclid=CjwKCAiAxea5BhBeEiwAh4t5K2poPd-WtFHvrB6_1SD4ON3P9iVZ4SIZSqrdqDDWPsc7A5bqGwFiVBoCPfcQAvD_BwE) and click Download. Open **Neo4j Desktop** after installation. Sign in or create a **Neo4j account** (required to activate the software).\nOnce logged in, create a **New Project** :\n * Click on the `**+**`button in the top-left corner.\n * Name your project (e.g., \"Wiki Movie KG\").\n\n\nInside your project, click on `**Add Database**`. Select**Local DBMS** and click **Create a Local Graph**.\nConfigure your database:\n * **Name** : Enter a name (e.g., `neo4j`).\n * **Password** : Set a password (e.g., `ilovemovies`). Remember this password for later.\n\n\nClick **Create** to initialize the database.\nNext, let's move to our LLM. The preferred way of running this notebook is using Ollama. **Ollama** is a locally hosted LLM solution that lets you download and set up LLMs on your laptop really really easily. It supports many open-source LLMs including Llama by Meta and Gemma by Google. I prefer this step as running your local LLM is free (excluding the degradation/energy cost of your laptop), and private, and it's just more exciting.\nTo download Ollama visit [Ollama's official website](https://ollama.ai/), and download the installer for your operating system. Open the **Ollama** application after installation.\nOpen a terminal and use the following command to list available models:\nbash\n\n```\nollama list\n```\n\nInstall and run a model. we will be using `qwen2.5-coder:latest` , which is a 7B Language Model fine-tuned on code tasks.\nbash\n\n```\nollama run qwen2.5-coder:latest\n```\n\nVerify the installation:\nbash\n\n```\nollama list\n```\n\nYou should now see:\nbash\n\n```\nqwen2.5-coder:latest\n```\n\nAnother free alternative is Gemini by Google, which lets us run 1500 requests per day. This solution actually outperforms the previous one since we are using a bigger and more powerful model. However, you may hit the limit depending on how many times you execute the script in a day.\nTo get a free API Key with Gemini, visit the [website](https://www.googleadservices.com/pagead/aclk?sa=L&ai=DChcSEwiL4OC_neSJAxV-cEcBHSPmEjUYABAAGgJxdQ&co=1&ase=2&gclid=CjwKCAiAxea5BhBeEiwAh4t5K-5Er62K62POougv5RZFpAglDFSVg43SrYLQM6MWWPP9LuJpTrnjTRoCNo4QAvD_BwE&ei=SlU6Z8KoOdWt5NoP0oio6QY&ohost=www.google.com&cid=CAESVuD2RgfXSVKvkdgIqHdVxFkchLNjCfya9TRqTM6SsW3mXjsq4CFoeiUD5rFUM47GMTu1RN9W0JE9DzrlzBDI75iT7oDTASAPGQU8SpT5_Z6M101LwjzM&sig=AOD64_3PJYSJ7hFfGD_flW0QfFen0Mypnw&q&sqi=2&nis=4&adurl&ved=2ahUKEwjC9dq_neSJAxXVFlkFHVIEKm0Q0Qx6BAgLEAE) and click \"Get an API Key\". Then follow the instructions and copy the API key generated. We will use it in a moment.\n### Graph Builder from Scratch\nLet's start by importing a few libraries required for the project:\npython\n\n```\n# Type hints\nfrom typing import Any, Dict, List, Tuple\n\n# Standard library\nimport ast\nimport logging\nimport re\nimport warnings\n\n# Third-party packages - Data manipulation\nimport pandas as pd\nfrom tqdm import tqdm\n\n# Third-party packages - Environment & Database\nfrom dotenv import load_dotenv\nfrom neo4j import GraphDatabase\n\n# Third-party packages - Error handling & Retry logic\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\n# Langchain - Core\nfrom langchain.chains import GraphCypherQAChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain_core.documents import Document\n\n# Langchain - Models & Connectors\nfrom langchain_google_genai import ChatGoogleGenerativeAI, GoogleGenerativeAI\nfrom langchain_ollama.llms import OllamaLLM\n\n# Langchain - Graph & Experimental\nfrom langchain_community.graphs import Neo4jGraph\nfrom langchain_experimental.graph_transformers import LLMGraphTransformer\n\n# Suppress warnings\nwarnings.filterwarnings('ignore')\n\n# Load environment variables\nload_dotenv()\n```\n\nAs you can see, LangChain doesn't do a good job with the code organization, resulting in quite a few lines of import. Let's break down the libraries we are importing:\n 1. `**os**`**and**`**dotenv**`: Help us manage environment variables (like database credentials).\n 2. `**pandas**`: Used to handle and process the movie dataset.\n 3. `**neo4j**`: This library connects Python to the Neo4j graph database.\n 4. `**langchain**`: Provides tools to work with Language Models (LLMs) and graphs.\n 5. `**tqdm**`: Adds a nice UI to print statements. We will use it to show a progress bar in loops, so we know how much processing is left.\n 6. `**warnings**`: Suppresses unnecessary warnings for a cleaner output.\n\n\nWe load the movie dataset, which contains information about 34,886 movies from around the world. The dataset is publicly available on [Kaggle](https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots) ([License: CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)). However, if you cloned my GitHub repo, the dataset will already be present in the data folder:\npython\n\n```\nmovies = pd.read_csv('data/wiki_movies.csv') # adjust the path if you manually downloaded the dataset\nmovies.head()\n```\n\nHere, we can see the following features:\n * _Release Year:_ The year in which the movie was released\n * _Title:_ The movie title\n * _Origin/Ethnicity: The o_rigin of the movie (i.e. American, Bollywood, Tamil, etc.)\n * _Director:_ Director(s)\n * _Plot:_ Main actor and actresses\n * _Genre:_ Movie Genre(s)\n * _Wiki Page:_ URL of the Wikipedia page from which the plot description was scraped\n * _Plot:_ Long-form description of movie plot\n\n\nBy looking at these features, we could quickly come up with some of the labels and relationships we would like to see in our KG. Since this is a movie dataset, a movie would be one of them. Moreover, we may be interested in querying for specific actors and directors. Hence, we end up with three labels for our nodes: Movie, Actor, and Director. Of course, we could include more labels. However, let's stop here for the sake of simplicity.\nFor the sake of simplicity, let's clean a little bit this dataset, and extract only the first 1000 rows:\npython\n\n```\ndef clean_data(df: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Clean and preprocess DataFrame.\n\n Args:\n data: Input DataFrame\n\n Returns:\n Cleaned DataFrame\n \"\"\"\n df.drop([\"Wiki Page\"], axis=1, inplace=True)\n\n # Drop duplicates\n df = df.drop_duplicates(subset='Title', keep='first')\n\n # Get object columns\n col_obj = df.select_dtypes(include=[\"object\"]).columns\n\n # Clean string columns\n for col in col_obj:\n # Strip whitespace\n df[col] = df[col].str.strip()\n\n # Replace unknown/empty values\n df[col] = df[col].apply(\n lambda x: None if pd.isna(x) or x.lower() in [\"\", \"unknown\"] \n else x.capitalize()\n )\n\n # Drop rows with any null values\n df = df.dropna(how=\"any\", axis=0)\n\n return df\n\nmovies = clean_data(movies).head(1000)\nmovies.head()\n```\n\nHere, we are dropping the `Wiki Page` column, which contains the link to the Wikipedia page. However, feel free to keep it, as this could be a property for the `Movie` nodes. Next, we drop all the duplicates by title, and we clean all the _string_ (`object`) columns. Lastly, we keep only the first 1000 movies.\nSince our knowledge graph will be hosted on Neo4j, let's set up a helper class to establish the connection and provide useful methods:\npython\n\n```\nclass Neo4jConnection:\n def __init__(self, uri, user, password):\n self.driver = GraphDatabase.driver(uri, auth=(user, password))\n\n def close(self):\n self.driver.close()\n print(\"Connection closed\")\n\n def reset_database(self):\n with self.driver.session() as session:\n session.run(\"MATCH (n) DETACH DELETE n\")\n print(\"Database resetted successfully!\")\n\n def execute_query(self, query, parameters=None):\n with self.driver.session() as session:\n result = session.run(query, parameters or {})\n return [record for record in result]\n```\n\nIn the initialization (`__init__`), we set up the connection to the Neo4j database using the database URL (`uri`), username, and password. We will pass these variables later when we initialize the class.\nThe method`**close**`terminates the connection to the database.\n`**reset_database**`deletes all nodes and relationships in the database using the Cypher command`MATCH (n) DETACH DELETE n`.\n`**execute_query**`runs a given query (like adding a movie or fetching relationships) and returns the results.\nNext, let's connect to the database using the helper class:\npython\n\n```\nuri = \"bolt://localhost:7687\"\nuser = \"neo4j\"\npassword = \"your_password_here\"\nconn = Neo4jConnection(uri, user, password)\nconn.reset_database()\n```\n\nBy default `uri`, and `user` will match the ones provided above. As for `password` , this will be the one you defined while creating the database. Moreover, let's `**reset_database**`to ensure we start with a clean slate by removing any existing data.\nIf you encounter any errors related to APOC not being installed in your database, go to Neo4j -> click on the database -> Plugins -> Install APOC:\n![Neo4j Desktop - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/17BRr9TlXyJFntU_eioBBGg.png)\nNeo4j Desktop - Image by Author\nWe now need to take each movie from the dataset and turn it into a node in our graph. In this section, we will do it manually, whereas in the next sections, we will leverage an LLM to do it for us.\npython\n\n```\ndef parse_number(value: Any, target_type: type) -> Optional[float]:\n \"\"\"Parse string to number with proper error handling.\"\"\"\n if pd.isna(value):\n return None\n try:\n cleaned = str(value).strip().replace(',', '')\n return target_type(cleaned)\n except (ValueError, TypeError):\n return None\n\ndef clean_text(text: str) -> str:\n \"\"\"Clean and normalize text fields.\"\"\"\n if pd.isna(text):\n return \"\"\n return str(text).strip().title()\n```\n\nLet's create two short functions - `**parse_number**`**and**`**clean_text**`- to convert data into numbers for numerical columns, and properly format the text columns. If the conversion fails (e.g., if the value is empty), they return`None` in case of a numerical column, and an empty string for object columns.\nNext, let's create a to iteratively load data into our KG:\nphp\n\n```\ndef load_movies_to_neo4j(movies_df: pd.DataFrame, connection: GraphDatabase) -> None:\n \"\"\"Load movie data into Neo4j with progress tracking and error handling.\"\"\"\n\n logger = logging.getLogger(__name__)\n logger.setLevel(logging.INFO)\n\n # Query templates\n MOVIE_QUERY = \"\"\"\n MERGE (movie:Movie {title: $title})\n SET movie.year = $year,\n movie.origin = $origin,\n movie.genre = $genre,\n movie.plot = $plot\n \"\"\"\n\n DIRECTOR_QUERY = \"\"\"\n MATCH (movie:Movie {title: $title})\n MERGE (director:Director {name: $name})\n MERGE (director)-[:DIRECTED]->(movie)\n \"\"\"\n\n ACTOR_QUERY = \"\"\"\n MATCH (movie:Movie {title: $title})\n MERGE (actor:Actor {name: $name})\n MERGE (actor)-[:ACTED_IN]->(movie)\n \"\"\"\n\n # Process each movie\n for _, row in tqdm(movies_df.iterrows(), total=len(movies_df), desc=\"Loading movies\"):\n try:\n # Prepare movie parameters\n movie_params = {\n \"title\": clean_text(row[\"Title\"]),\n \"year\": parse_number(row[\"Release Year\"], int),\n \"origin\": clean_text(row[\"Origin/Ethnicity\"]),\n \"genre\": clean_text(row[\"Genre\"]),\n \"plot\": str(row[\"Plot\"]).strip()\n }\n\n # Create movie node\n connection.execute_query(MOVIE_QUERY, parameters=movie_params)\n\n # Process directors\n for director in str(row[\"Director\"]).split(\" and \"):\n director_params = {\n \"name\": clean_text(director),\n \"title\": movie_params[\"title\"]\n }\n connection.execute_query(DIRECTOR_QUERY, parameters=director_params)\n\n # Process cast\n if pd.notna(row[\"Cast\"]):\n for actor in row[\"Cast\"].split(\",\"):\n actor_params = {\n \"name\": clean_text(actor),\n \"title\": movie_params[\"title\"]\n }\n connection.execute_query(ACTOR_QUERY, parameters=actor_params)\n\n except Exception as e:\n logger.error(f\"Error loading {row['Title']}: {str(e)}\")\n continue\n\n logger.info(\"Finished loading movies to Neo4j\")\n```\n\nTwo important keywords to know to understand the Cypher query above are `MERGE` and `SET`.\n`**MERGE**`ensures the node or relationship exists; if not, it creates it. Therefore, it combines both the`MATCH` , and `CREATE` clauses, where `MATCH` allows us to search within the graph for a certain structure, and `CREATE` to create nodes and relationships. Therefore, `MERGE` first check if the node/edge we are creating doesn't exist, if it doesn't then it creates it.\nIn the function above we use `MERGE` to create a node for every Movie, Director, and Actor. In particular, since we have features for actors (Star1, Star2, Star3, and Star4), we create an Actor node for each column.\nNext, we create two relationships: one from Director to Movie (`DIRECTED`), and from Actor to Movie (`ACTED_IN`).\n`**SET**`is used to update node or edge properties. In this case, we are providing the Movie node with Year, Rating, Genre, Runtime, and Overview; and the Director, and Actor with a Name property.\nAlso, note that we are using the special symbol `$` to define a parameter.\nNext, let's call the function and load all the movies:\npython\n\n```\nload_movies_to_neo4j(movies, conn)\n```\n\nIt will take approximately a minute to load all 1000 movies.\nOnce, the execution is completed, let's run a Cypher query to check if the movies are uploaded correctly：\npython\n\n```\nquery = \"\"\"\nMATCH (m:Movie)-[:ACTED_IN]-(a:Actor)\nRETURN m.title, a.name\nLIMIT 10;\n\"\"\"\nresults = conn.execute_query(query)\nfor record in results:\n print(record)\n```\n\nThis query should now look familiar. We used `**MATCH**`to find patterns in the graph.\n * `(m:Movie)`: Matches nodes labeled \"Movie\".\n * `[:ACTED_IN]`: Matches relationships of type \"ACTED_IN\".\n * `(a:Actor)`: Matches nodes labeled \"Actor\".\n\n\nAlso, we used `**RETURN**`to specify what to display—in this case, movie titles and actor names, and`**LIMIT**`to restrict the result to the first 10 matches.\nYou should get an output similar to this:\nbash\n\n```\n[,\n ,\n ,\n ,\n ,\n ,\n ,\n ,\n ,\n ]\n```\n\nOk, this list of 10 records representing movies and actors confirms that the movies have been uploaded on the KG.\nNext, let's see the actual graph. Switch to Neo4j Desktop, select the Movie database you created for this exercise, and click open with Neo4j Browser. This will open a new tab where you will able to run Cypher queries. Then, run the following query:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nYou should now see something like this:\n![Manual Knowledge Graph - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/193zzEUz9CBIve3oYHPcaRA-scaled.jpeg)\nManual Knowledge Graph - Image by Author\nQuite cool right?\nHowever, this did require some time to explore the dataset, do some cleaning, and manually write the Cypher queries. But come on, it's the Chat GPT era, of course, we don't need to do that anymore unless you want it. Multiple approaches are showing several ways to automate this process. In the next section, we will create a basic one leveraging an LLM.\n### Creating a Custom LLM to Automate Data Upload\nIn this section, we create a custom process where an LLM automatically generates node definitions, relationships, and Cypher queries based on the dataset. This approach could be used on other Dataframes as well as automatically recognize the schema. However, consider that it won't match the performances of modern solutions, like LLMGraphTransformer from LangChain, which we will cover in the next section. Instead, use this section to understand a possible \"from-scratch\" workflow, to get creative, and later design your own `Graph-Builder`. Indeed, if there is one main limitation of the current **SOTA** (_State-Of-The-Art_) approaches, that's being highly sensitive to the nature of data, and patterns. Therefore, being able to think outside the box is incredibly important in order to either design your Graph-RAG framework from scratch or being able to adapt an existing SOTA Graph-RAG to your needs.\nNow, let's get into it, and set up the LLM we will be using for this exercise. You could use any LLM supported by LangChain for this exercise, as long as it's enough powerful to match decent performances.\nTwo free approaches are Gemini, which is free up to 1500 requests per day using their `Gemini Flash` model, and `Ollama`, which lets you easily download open-source models on your laptop and set up an API that you can easily call using LangChain. I tested the notebook with both Gemini, and custom Ollama models, and although Gemini guarantees superior performances, I would highly recommend going with Ollama for learning purposes, as playing around with \"your own\" LLM is just more cool.\nIn the Ollama example, we will be using `qwen2.5-coder 7B`, which is fine-tune on Code-specific tasks, and has impressive performance on code generation, reasoning, and fixing. Based on your memory availability, and laptop performance, you could download the 14B, or 32B version, which would guarantee higher performance.\nLet's initialize the model:\npython\n\n```\n# llm = GoogleGenerativeAI(model=\"gemini-1.5-flash\", google_api_key=api_key) # if you are using Google API\nllm = OllamaLLM(model=\"qwen2.5-coder:latest\")\n```\n\nIf you chose Gemini as your solution, uncomment the first line of code, and comment on the second one. Also, if you chose Gemini, remember to provide the API Key.\nLet's start by extracting the structure of the dataset and defining the nodes and their properties:\npython\n\n```\nnode_structure = \"n\".join([\n f\"{col}: {', '.join(map(str, movies[col].unique()[:3]))}...\" \n for col in movies.columns\n])\nprint(node_structure)\n```\n\nFor each column in the dataset (e.g., `Genre`, `Director`), we display a few sample values. This gives the LLM an understanding of the data format and typical values for each column.\nbash\n\n```\nRelease Year: 1907, 1908, 1909...\nTitle: Daniel boone, Laughing gas, The adventures of dollie...\nOrigin/Ethnicity: American...\nDirector: Wallace mccutcheon and ediwin s. porter, Edwin stanton porter, D. w. griffith...\nCast: William craven, florence lawrence, Bertha regustus, edward boulden, Arthur v. johnson, linda arvidson...\nGenre: Biographical, Comedy, Drama...\nPlot: Boone's daughter befriends an indian maiden as boone and his companion start out on a hunting expedition. while he is away, boone's cabin is attacked by the indians, who set it on fire and abduct boone's daughter. boone returns, swears vengeance, then heads out on the trail to the indian camp. his daughter escapes but is chased. the indians encounter boone, which sets off a huge fight on the edge of a cliff. a burning arrow gets shot into the indian camp. boone gets tied to the stake and tortured. the burning arrow sets the indian camp on fire, causing panic. boone is rescued by his horse, and boone has a knife fight in which he kills the indian chief.[2], The plot is that of a black woman going to the dentist for a toothache and being given laughing gas. on her way walking home, and in other situations, she can't stop laughing, and everyone she meets \"catches\" the laughter from her, including a vendor and police officers., On a beautiful summer day a father and mother take their daughter dollie on an outing to the river. the mother refuses to buy a gypsy's wares. the gypsy tries to rob the mother, but the father drives him off. the gypsy returns to the camp and devises a plan. they return and kidnap dollie while her parents are distracted. a rescue crew is organized, but the gypsy takes dollie to his camp. they gag dollie and hide her in a barrel before the rescue party gets to the camp. once they leave the gypsies and escapes in their wagon. as the wagon crosses the river, the barrel falls into the water. still sealed in the barrel, dollie is swept downstream in dangerous currents. a boy who is fishing in the river finds the barrel, and dollie is reunited safely with her parents...\n```\n\n**Generating Nodes** Next, we use an L**LM prompt template** to instruct the model on how to extract nodes and their properties. Let's first take a look at what the whole code looks like:\npython\n\n```\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\ndef validate_node_definition(node_def: Dict) -> bool:\n \"\"\"Validate node definition structure\"\"\"\n if not isinstance(node_def, dict):\n return False\n return all(\n isinstance(v, dict) and all(isinstance(k, str) for k in v.keys())\n for v in node_def.values()\n )\n\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef get_node_definitions(chain, structure: str, example: Dict) -> Dict[str, Dict[str, str]]:\n \"\"\"Get node definitions with retry logic\"\"\"\n try:\n # Get response from LLM\n response = chain.invoke({\"structure\": structure, \"example\": example})\n\n # Parse response\n node_defs = ast.literal_eval(response)\n\n # Validate structure\n if not validate_node_definition(node_defs):\n raise ValueError(\"Invalid node definition structure\")\n\n return node_defs\n\n except (ValueError, SyntaxError) as e:\n logger.error(f\"Error parsing node definitions: {e}\")\n raise\n\n# Updated node definition template\nnode_example = {\n \"NodeLabel1\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel2\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel3\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n}\n\ndefine_nodes_prompt = PromptTemplate(\n input_variables=[\"example\", \"structure\"],\n template=(\"\"\"\n Analyze the dataset structure below and extract the entity labels for nodes and their properties.n\n The node properties should be based on the dataset columns and their values.n\n Return the result as a dictionary where the keys are the node labels and the values are the node properties.nn\n Example: {example}nn\n\n Dataset Structure:n{structure}nn\n\n Make sure to include all the possible node labels and their properties.n\n If a property can be its own node, include it as a separate node label.n\n Please do not report triple backticks to identify a code block, just return the list of tuples.n\n Return only the dictionary containing node labels and properties, and don't include any other text or quotation.\n\n \"\"\"\n ),\n)\n\n# Execute with error handling\ntry:\n node_chain = define_nodes_prompt | llm\n\n node_definitions = get_node_definitions(node_chain, structure=node_structure, example=node_example)\n logger.info(f\"Node Definitions: {node_definitions}\")\nexcept Exception as e:\n logger.error(f\"Failed to get node definitions: {e}\")\n raise\n```\n\nHere, we first set up logging using the `logging` library, which is a Python module to track events during execution (like errors or status updates):\npython\n\n```\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n```\n\nWe use `**basicConfig**`to configure logging to display messages of level`INFO` or higher, and initiate the `**logger**`instance, which we will use throughout the code to log messages.\nThis step is not really required, and you could replace it with just print statements. However, it's a good engineering practice.\nNext, we create a function to validate the nodes generated by the LLM:\npython\n\n```\ndef validate_node_definition(node_def: Dict) -> bool:\n \"\"\"Validate node definition structure\"\"\"\n if not isinstance(node_def, dict):\n return False\n return all(\n isinstance(v, dict) and all(isinstance(k, str) for k in v.keys())\n for v in node_def.values()\n )\n```\n\nThe input of the function is a dictionary, where keys are node labels (e.g., `Movie`) and values are dictionaries of properties (e.g., `title`, `year`).\nFirst, the function checks if `node_def` is a dictionary, and verifies that each value in the dictionary is also a dictionary and that all keys inside these dictionaries are strings. Then it returns `**True**`if the structure is valid.\nNext, we create a function to invoke the LLM chain and actually generate the nodes:\npython\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef get_node_definitions(chain, structure: str, example: Dict) -> Dict[str, Dict[str, str]]:\n \"\"\"Get node definitions with retry logic\"\"\"\n try:\n # Get response from LLM\n response = chain.invoke({\"structure\": structure, \"example\": example})\n\n # Parse response\n node_defs = ast.literal_eval(response)\n\n # Validate structure\n if not validate_node_definition(node_defs):\n raise ValueError(\"Invalid node definition structure\")\n\n return node_defs\n```\n\nIf you are not familiar with decorators, you may be wondering what `@retry(...)` is doing. Look at this as a wrapper function that surrounds the actual `get_node_definitions` function. In this case, we call the `retry` decorator, which automatically retries the function if an error occurs.\n * `**stop_after_attempt(3)**`: Retries up to 3 times.\n * `**wait_exponential**`: Adds an increasing delay between retries (e.g., 4s, 8s, 16s).\n\n\nThe inputs for our functions are:\n * `**chain**`: The LangChain pipeline (prompt + LLM). We will define the chain later.\n * `**structure**`: The dataset structure (columns and sample values).\n * `**example**`: A sample node definition to guide the LLM.\n\n\nNext, `**chain.invoke**`sends the`structure` and `example` to the LLM and receives a response as a string. `**ast.literal_eval**`converts the string response into a Python dictionary.\nWe check if the parsed dictionary is in the correct format using `validate_node_definition`, which raises a `ValueError` if the structure is invalid.\npython\n\n```\nexcept (ValueError, SyntaxError) as e:\n logger.error(f\"Error parsing node definitions: {e}\")\n raise\n```\n\nIf the response cannot be parsed or validated, the error is logged, and the function raises an exception.\nNext, let's provide a prompt template to the LLM to guide it on the node generation task:\npython\n\n```\ndefine_nodes_prompt = PromptTemplate(\n input_variables=[\"example\", \"structure\"],\n template=(\"\"\"\n Analyze the dataset structure below and extract the entity labels for nodes and their properties.n\n The node properties should be based on the dataset columns and their values.n\n Return the result as a dictionary where the keys are the node labels and the values are the node properties.nn\n Example: {example}nn\n Dataset Structure:n{structure}nn\n Make sure to include all the possible node labels and their properties.n\n If a property can be its own node, include it as a separate node label.n\n Please do not report triple backticks to identify a code block, just return the list of tuples.n\n Return only the dictionary containing node labels and properties, and don't include any other text or quotation.\n \"\"\"),\n)\n```\n\nNote that we are providing the node structure defined at the beginning of this section, and an example of how to generate a dictionary of nodes:\npython\n\n```\nnode_example = {\n \"NodeLabel1\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel2\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n \"NodeLabel3\": {\"property1\": \"row['property1']\", \"property2\": \"row['property2']\"},\n}\n```\n\nIn the example, the keys are Node labels (e.g., `Movie`, `Director`), and the values are dictionaries of properties mapped to dataset columns (e.g., `row['property1']`).\nNext, let's execute the chain:\npython\n\n```\ntry:\n node_chain = define_nodes_prompt | llm\n node_definitions = get_node_definitions(node_chain, structure=node_structure, example=node_example)\n logger.info(f\"Node Definitions: {node_definitions}\")\nexcept Exception as e:\n logger.error(f\"Failed to get node definitions: {e}\")\n raise\n```\n\nIn LangChain we create a chain using the structure `prompt | llm | ...` , which combines the prompt template with the LLM, forming a pipeline. We use `**get_node_definitions**`to fetch and validate the node definitions.\nIf the process fails, the error is logged, and the program raises an exception.\nIf the process succeeds, it will generate something similar to this:\nbash\n\n```\nINFO:__main__:Node Definitions: {'Movie': {'Release Year': \"row['Release Year']\", 'Title': \"row['Title']\"}, 'Director': {'Name': \"row['Director']\"}, 'Cast': {'Actor': \"row['Cast']\"}, 'Genre': {'Type': \"row['Genre']\"}, 'Plot': {'Description': \"row['Plot']\"}}\n```\n\n**Generate Relationships** Once nodes are defined, we identify relationships between them. Again let's first take a look at what the whole code looks like:\npython\n\n```\nclass RelationshipIdentifier:\n \"\"\"Identifies relationships between nodes in a graph database.\"\"\"\n\n RELATIONSHIP_EXAMPLE = [\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel2\"),\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel3\"),\n (\"NodeLabel2\", \"RelationshipLabel\", \"NodeLabel3\"),\n ]\n\n PROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"example\"],\n template=\"\"\"\n Consider the following Dataset Structure:n{structure}nn\n\n Consider the following Node Definitions:n{node_definitions}nn\n\n Based on the dataset structure and node definitions, identify relationships (edges) between nodes.n\n Return the relationships as a list of triples where each triple contains the start node label, relationship label, and end node label, and each triple is a tuple.n\n Please return only the list of tuples. Please do not report triple backticks to identify a code block, just return the list of tuples.nn\n\n Example:n{example}\n \"\"\"\n)\n\n def __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n\n def validate_relationships(self, relationships: List[Tuple]) -> bool:\n \"\"\"Validate relationship structure.\"\"\"\n return all(\n isinstance(rel, tuple) and \n len(rel) == 3 and \n all(isinstance(x, str) for x in rel)\n for rel in relationships\n )\n\n @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def identify_relationships(self, structure: str, node_definitions: Dict) -> List[Tuple]:\n \"\"\"Identify relationships with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"structure\": structure, \n \"node_definitions\": str(node_definitions), \n \"example\": str(self.RELATIONSHIP_EXAMPLE)\n })\n\n relationships = ast.literal_eval(response)\n\n if not self.validate_relationships(relationships):\n raise ValueError(\"Invalid relationship structure\")\n\n self.logger.info(f\"Identified {len(relationships)} relationships\")\n return relationships\n\n except Exception as e:\n self.logger.error(f\"Error identifying relationships: {e}\")\n raise\n\n def get_relationship_types(self) -> List[str]:\n \"\"\"Extract unique relationship types.\"\"\"\n return list(set(rel[1] for rel in self.identify_relationships()))\n\n# Usage\nidentifier = RelationshipIdentifier(llm=llm)\nrelationships = identifier.identify_relationships(node_structure, node_definitions)\nprint(\"Relationships:\", relationships)\n```\n\nSince this code requires a few more operations the node generation, we organize the code in a class - `RelationshipIdentifier` - to encapsulate all the logic for relationship extraction, validation, and logging.\nWe use a similar logic, hence we provide a relationship example:\npython\n\n```\nRELATIONSHIP_EXAMPLE = [\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel2\"),\n (\"NodeLabel1\", \"RelationshipLabel\", \"NodeLabel3\"),\n (\"NodeLabel2\", \"RelationshipLabel\", \"NodeLabel3\"),\n]\n```\n\nHere, each relationship is a tuple with:\n * **Start Node Label** : The label of the source node (e.g., `Movie`).\n * **Relationship Label** : The connection type (e.g., `DIRECTED_BY`).\n * **End Node Label** : The label of the target node (e.g., `Director`).\n\n\nNext, we define the actual prompt template:\npython\n\n```\nPROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"example\"],\n template=\"\"\"\n Consider the following Dataset Structure:n{structure}nn\n\n Consider the following Node Definitions:n{node_definitions}nn\n\n Based on the dataset structure and node definitions, identify relationships (edges) between nodes.n\n Return the relationships as a list of triples where each triple contains the start node label, relationship label, and end node label, and each triple is a tuple.n\n Please return only the list of tuples. Please do not report triple backticks to identify a code block, just return the list of tuples.nn\n\n Example:n{example}\n \"\"\"\n)\n```\n\nIn this case, we have three input variables:\n * `**structure**`: The dataset structure, listing columns and sample values. We defined it at the beginning of the section.\n * `**node_definitions**`: A dictionary of node labels and their properties. These are the nodes generated by the LLM in the previous section.\n * `**example**`: Example relationships in tuple format.\n\n\nNext, we initialize the class with three attributes:\npython\n\n```\ndef __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n```\n\n * `**llm**`: The Language Model to process the prompt (e.g., GPT-3.5 Turbo).\n * `**logger**`: Optional; logs progress and errors (defaults to a standard logger if not provided).\n * `**self.chain**`: Combines the prompt template with the LLM to create a reusable pipeline.\n\n\nSimilarly to before, we create a method to validate the generated relationships:\npython\n\n```\ndef validate_relationships(self, relationships: List[Tuple]) -> bool:\n \"\"\"Validate relationship structure.\"\"\"\n return all(\n isinstance(rel, tuple) and \n len(rel) == 3 and \n all(isinstance(x, str) for x in rel)\n for rel in relationships\n )\n```\n\nThe method checks if each item is a tuple if each tuple contains exactly three elements, and if all elements are strings (e.g., node labels or relationship types). Lastly, it returns `TRUE` if the conditions are satisfied, otherwise `FALSE` .\nNext, we create a method to invoke the chain and generate the relationships:\npython\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\ndef identify_relationships(self, structure: str, node_definitions: Dict) -> List[Tuple]:\n \"\"\"Identify relationships with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"structure\": structure, \n \"node_definitions\": str(node_definitions), \n \"example\": str(self.RELATIONSHIP_EXAMPLE)\n })\n\n relationships = ast.literal_eval(response)\n\n if not self.validate_relationships(relationships):\n raise ValueError(\"Invalid relationship structure\")\n\n self.logger.info(f\"Identified {len(relationships)} relationships\")\n return relationships\n```\n\nWe use again the retry decorator to reattempt the LLM chain in case of failure and invoke the chain similarly to how we invoked it in the nodes generation.\nAdditionally, we use`**ast.literal_eval**`to convert the LLM's string output into a Python list and`validate_relationships` to ensure the output format is correct.\npython\n\n```\nexcept Exception as e:\n self.logger.error(f\"Error identifying relationships: {e}\")\n raise\n```\n\nIf the method fails, it logs the errors and retries the process up to 3 times.\nThe last method Returns unique relationship labels (e.g., `DIRECTED_BY`, `ACTED_IN`):\nrust\n\n```\ndef get_relationship_types(self) -> List[str]:\n \"\"\"Extract unique relationship types.\"\"\"\n return list(set(rel[1] for rel in self.identify_relationships()))\n```\n\nIt calls the`identify_relationships` method to get the list of relationships. Then, it extracts the second element (relationship label) from each tuple, uses `set` to remove duplicates, and converts the result back to a list.\nNow, it's finally time to generate the relationships:\npython\n\n```\nidentifier = RelationshipIdentifier(llm=llm)\nrelationships = identifier.identify_relationships(node_structure, node_definitions)\nprint(\"Relationships:\", relationships)\n```\n\nIf the LLM is successful within the 3 attempts it returns a list of relationships in tuple format similar to the following:\nbash\n\n```\nINFO:__main__:Identified 4 relationships\nRelationships: [('Movie', 'Directed By', 'Director'), ('Movie', 'Starring', 'Cast'), ('Movie', 'Has Genre', 'Genre'), ('Movie', 'Contains Plot', 'Plot')]\n```\n\n**Generate Cypher Queries** With nodes and relationships defined, we create Cypher queries to load them into Neo4j. The process follows a similar logic to both node generation and relationship generation. However, we define a couple more steps for validation, since the output generated will be used to load the data into our KG. Therefore, we need to maximize our chances of success. Let's first take a look at the whole code:\npython\n\n```\nclass CypherQueryBuilder:\n \"\"\"Builds Cypher queries for Neo4j graph database.\"\"\"\n\n INPUT_EXAMPLE = \"\"\"\n NodeLabel1: value1, value2\n NodeLabel2: value1, value2\n \"\"\"\n\n EXAMPLE_CYPHER = example_cypher = \"\"\"\n CREATE (n1:NodeLabel1 {property1: \"row['property1']\", property2: \"row['property2']\"})\n CREATE (n2:NodeLabel2 {property1: \"row['property1']\", property2: \"row['property2']\"})\n CREATE (n1)-[:RelationshipLabel]->(n2);\n \"\"\"\n\n PROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"relationships\", \"example\"],\n template=\"\"\"\n Consider the following Node Definitions:n{node_definitions}nn\n Consider the following Relationships:n{relationships}nn\n Generate Cypher queries to create nodes and relationships using the node definitions and relationships below. Remember to replace the placeholder values with actual data from the dataset.n\n Include all the properties in the Node Definitions for each node as defined and create relationships.n\n Return a single string with each query separated by a semicolon.n\n Don't include any other text or quotation marks in the response.n\n Please return only the string containing Cypher queries. Please do not report triple backticks to identify a code block.nn\n\n Example Input:n{input}nn\n\n Example Output Cypher query:n{cypher}\n \"\"\"\n)\n\n def __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n # self.chain = LLMChain(llm=llm, prompt=self.PROMPT_TEMPLATE)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n\n def validate_cypher_query(self, query: str) -> bool:\n \"\"\"Validate Cypher query syntax using LLM and regex patterns.\"\"\"\n\n VALIDATION_PROMPT = PromptTemplate(\n input_variables=[\"query\"],\n template=\"\"\"\n Validate this Cypher query and return TRUE or FALSE:\n\n Query: {query}\n\n Rules to check:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n Return only TRUE if query is valid, FALSE if invalid.\n \"\"\"\n )\n\n try:\n # Basic pattern validation\n basic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n ])\n\n if not basic_valid:\n return False\n\n # LLM validation\n validation_chain = VALIDATION_PROMPT | self.llm\n result = validation_chain.invoke({\"query\": query})\n\n # Parse result\n is_valid = \"TRUE\" in result.upper()\n\n if not is_valid:\n self.logger.warning(f\"LLM validation failed for query: {query}\")\n\n return is_valid\n\n except Exception as e:\n self.logger.error(f\"Validation error: {e}\")\n return False\n\n def sanitize_query(self, query: str) -> str:\n \"\"\"Sanitize and format Cypher query.\"\"\"\n return (query\n .strip()\n .replace('n', ' ')\n .replace(' ', ' ')\n .replace(\"'row[\", \"row['\")\n .replace(\"]'\", \"']\"))\n\n @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def build_queries(self, node_definitions: Dict, relationships: List) -> str:\n \"\"\"Build Cypher queries with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"node_definitions\": str(node_definitions),\n \"relationships\": str(relationships),\n \"input\": self.INPUT_EXAMPLE,\n \"cypher\": self.EXAMPLE_CYPHER\n })\n\n # Get response inside triple backticks\n if '```' in response:\n response = response.split('```')[1]\n\n # Sanitize response\n queries = self.sanitize_query(response)\n\n # Validate queries\n if not self.validate_cypher_query(queries):\n raise ValueError(\"Invalid Cypher query syntax\")\n\n self.logger.info(\"Successfully generated Cypher queries\")\n return queries\n\n except Exception as e:\n self.logger.error(f\"Error building Cypher queries: {e}\")\n raise\n\n def split_queries(self, queries: str) -> List[str]:\n \"\"\"Split combined queries into individual statements.\"\"\"\n return [q.strip() for q in queries.split(';') if q.strip()]\n\n# Usage\nbuilder = CypherQueryBuilder(llm=llm)\ncypher_queries = builder.build_queries(node_definitions, relationships)\nprint(\"Cypher Queries:\", cypher_queries)\n```\n\nWe provide a prompt template to help the LLM:\npython\n\n```\nPROMPT_TEMPLATE = PromptTemplate(\n input_variables=[\"structure\", \"node_definitions\", \"relationships\", \"example\"],\n template=\"\"\"\n Consider the following Node Definitions:n{node_definitions}nn\n Consider the following Relationships:n{relationships}nn\n Generate Cypher queries to create nodes and relationships using the node definitions and relationships below. Remember to replace the placeholder values with actual data from the dataset.n\n Include all the properties in the Node Definitions for each node as defined and create relationships.n\n Return a single string with each query separated by a semicolon.n\n Don't include any other text or quotation marks in the response.n\n Please return only the string containing Cypher queries. Please do not report triple backticks to identify a code block.nn\n\n Example Input:n{input}nn\n\n Example Output Cypher query:n{cypher}\n \"\"\"\n)\n```\n\nNow we provide four inputs to the prompt:\n * `**structure**`: Dataset structure for context.\n * `**node_definitions**`: Generated Nodes and their properties.\n * `**relationships**`: Generated Edges between nodes.\n * `**example**`: Example queries for formatting reference.\n\n\npython\n\n```\ndef __init__(self, llm: Any, logger: logging.Logger = None):\n self.llm = llm\n self.logger = logger or logging.getLogger(__name__)\n self.chain = self.PROMPT_TEMPLATE | self.llm\n```\n\nWe initialize the class in the same fashion as the relationship class.\nNext, we define a validation method to check the generated output:\npython\n\n```\ndef validate_cypher_query(self, query: str) -> bool:\n \"\"\"Validate Cypher query syntax using LLM and regex patterns.\"\"\"\n\n VALIDATION_PROMPT = PromptTemplate(\n input_variables=[\"query\"],\n template=\"\"\"\n Validate this Cypher query and return TRUE or FALSE:\n\n Query: {query}\n\n Rules to check:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n Return only TRUE if query is valid, FALSE if invalid.\n \"\"\"56\n )\n\n try:\n # Basic pattern validation\n basic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n ])\n\n if not basic_valid:\n return False\n\n # LLM validation\n validation_chain = VALIDATION_PROMPT | self.llm\n result = validation_chain.invoke({\"query\": query})\n\n # Parse result\n is_valid = \"TRUE\" in result.upper()\n\n if not is_valid:\n self.logger.warning(f\"LLM validation failed for query: {query}\")\n\n return is_valid\n\n except Exception as e:\n self.logger.error(f\"Validation error: {e}\")\n return False\n```\n\nThis method does two validation steps. First a basic validation with regular expressions:\npython\n\n```\nbasic_valid = all(re.search(pattern, query) for pattern in [\n r'CREATE (', \n r'{.*?}', \n r')-[:.*?]->'\n])\nif not basic_valid:\n return False\n```\n\nThis ensures the query contains essential Cypher syntax:\n * `**CREATE**`: Ensures nodes and relationships are being created.\n * *_`{._?}`**: Ensures properties are included.\n * *_`-[:._?]->`**: Ensures relationships are correctly formatted.\n\n\nThen, it performs an advanced validation with LLM:\npython\n\n```\nvalidation_chain = VALIDATION_PROMPT | self.llm\nresult = validation_chain.invoke({\"query\": query})\nis_valid = \"TRUE\" in result.upper()\n```\n\nThe validation is specified in the prompt where we ask the LLM to make sure that we have:\n 1. Valid CREATE statements\n 2. Proper property formatting\n 3. Valid relationship syntax\n 4. No missing parentheses\n 5. Valid property names\n 6. Valid relationship types\n\n\nEven though we should be in a good state as of now, let's add a method, to further sanitize the generated output:\nscss\n\n```\ndef sanitize_query(self, query: str) -> str:\n \"\"\"Sanitize and format Cypher query.\"\"\"\n return (query\n .strip()\n .replace('n', ' ')\n .replace(' ', ' ')\n .replace(\"'row[\", \"row['\")\n .replace(\"]'\", \"']\"))\n```\n\nIn particular, we are removing unnecessary spaces, as well as line breaks (`n`), and fix potential formatting issues with dataset references (e.g., `row['property1']`).\nPlease consider updating this method based on the model you are using. Smaller models will likely require more sanitization.\nNext, we define a query invocation method:\nless\n\n```\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))\n def build_queries(self, node_definitions: Dict, relationships: List) -> str:\n \"\"\"Build Cypher queries with retry logic.\"\"\"\n try:\n response = self.chain.invoke({\n \"node_definitions\": str(node_definitions),\n \"relationships\": str(relationships),\n \"input\": self.INPUT_EXAMPLE,\n \"cypher\": self.EXAMPLE_CYPHER\n })\n\n # Get response inside triple backticks\n if '```' in response:\n response = response.split('```')[1]\n\n # Sanitize response\n queries = self.sanitize_query(response)\n\n # Validate queries\n if not self.validate_cypher_query(queries):\n raise ValueError(\"Invalid Cypher query syntax\")\n\n self.logger.info(\"Successfully generated Cypher queries\")\n return queries\n\n except Exception as e:\n self.logger.error(f\"Error building Cypher queries: {e}\")\n raise\n```\n\nThis method works similarly to the one in the relationship builder class, with the only addition of:\npython\n\n```\nif '```' in response:\n response = response.split('```')[1]\n```\n\nHere, the LLM may provide an additional markdown format to specify it's a code block. If this is present in the LLM's response, we only retrieve the code within the triple backticks.\nNext, we define a method to break a single string of Cypher queries into individual statements:\npython\n\n```\ndef split_queries(self, queries: str) -> List[str]:\n \"\"\"Split combined queries into individual statements.\"\"\"\n return [q.strip() for q in queries.split(';') if q.strip()]\n```\n\nFor example, this Cypher query:\nsql\n\n```\nCREATE (n1:Movie {title: \"Inception\"}); CREATE (n2:Director {name: \"Nolan\"});\n```\n\nThis will turn into this:\nbash\n\n```\n[\"CREATE (n1:Movie {title: 'Inception'})\", \"CREATE (n2:Director {name: 'Nolan'})\"]\n```\n\nThis will be useful so we can iterate over a list of queries.\nLastly, we initialize the class and generate the Cypher queries:\npython\n\n```\nbuilder = CypherQueryBuilder(llm=llm)\ncypher_queries = builder.build_queries(node_definitions, relationships)\nprint(\"Cypher Queries:\", cypher_queries)\n```\n\nIn case of success, the output will look like this:\nbash\n\n```\nINFO:__main__:Successfully generated Cypher queries\nCypher Queries: CREATE (m:Movie {Release_Year: \"row['Release Year']\", Title: \"row['Title']\"}) CREATE (d:Director {Name: \"row['Director']\"}) CREATE (c:Cast {Actor: \"row['Cast']\"}) CREATE (g:Genre {Type: \"row['Genre']\"}) CREATE (p:Plot {Description: \"row['Plot']\"}) CREATE (m)-[:Directed_By]->(d) CREATE (m)-[:Starring]->(c) CREATE (m)-[:Has_Genre]->(g) CREATE (m)-[:Contains_Plot]->(p)\n```\n\nFinally, we iterate over the dataset and execute the generated Cypher queries for each row.\npython\n\n```\nlogs = \"\"\ntotal_rows = len(df)\n\ndef sanitize_value(value):\n if isinstance(value, str):\n return value.replace('\"', '')\n return str(value)\n\nfor index, row in tqdm(df.iterrows(), \n total=total_rows,\n desc=\"Loading data to Neo4j\",\n position=0,\n leave=True):\n\n # Replace placeholders with actual values\n cypher_query = cypher_queries\n for column in df.columns:\n cypher_query = cypher_query.replace(\n f\"row['{column}']\", \n f'{sanitize_value(row[column])}'\n )\n\n try:\n # Execute query and update progress\n conn.execute_query(cypher_query)\n except Exception as e:\n logs += f\"Error on row {index+1}: {str(e)}n\"\n```\n\nNote that we define an empty string variable `logs` which we will use to capture potential failures. Also, we add a sanitize function to pass to each value in the row input:\npython\n\n```\ndef sanitize_value(value):\n if isinstance(value, str):\n return value.replace('\"', '')\n return str(value)\n```\n\nThis will prevent strings containing double quotes from breaking the query syntax.\nNext, we loop over the dataset:\npython\n\n```\nfor index, row in tqdm(df.iterrows(), \n total=total_rows,\n desc=\"Loading data to Neo4j\",\n position=0,\n leave=True):\n\n # Replace placeholders with actual values\n cypher_query = cypher_queries\n for column in df.columns:\n cypher_query = cypher_query.replace(\n f\"row['{column}']\", \n f'{sanitize_value(row[column])}'\n )\n\n try:\n # Execute query and update progress\n conn.execute_query(cypher_query)\n except Exception as e:\n logs += f\"Error on row {index+1}: {str(e)}n\"\n```\n\nAs I mentioned at the beginning of the exercise, we use `tqdm` to add a nice-looking progress bar to visualize how many rows have been processed. We pass `**df.iterrows()**`to iterate through the DataFrame, providing the index and the row data.`**total=total_rows**`is used by`tqdm` to calculate progress. We add`**desc=\"Loading data to Neo4j\"**`to provide a label for the progress bar. Lastly,`**position=0, leave=True**`ensures the progress bar stays visible in the console.\nNext, we replace placeholders like `row['column_name']` with the actual dataset values passing each value in the `sanitize_value` function, and execute the query.\nLet's check if our dataset is now uploaded. Switch to the Neo4j browser, and run the following Cypher query:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nIn my case, the LLM generated the following graph:\n![Custom LLM generated Knowledge Graph - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/17HnUIfVUKhqKklQPdISaXQ-scaled.jpeg)\nCustom LLM generated Knowledge Graph - Image by Author\nThis is pretty similar to the knowledge graph we uploaded manually. Not bad for a Naïve LLM right? Even though that required quite some code, we can now reuse it for multiple datasets, and more importantly, we can use it as a base to create more complex LLM graph builders.\nIn our example, we haven't helped our LLM by providing entities, relationships, and properties for both. However, consider adding them as examples to increase the performance of the LLM. Moreover, modern approaches leverage the chain of thought to come up with additional nodes and relationships, this makes the model sequentially reason over the output to further improve it. Another strategy can be providing samples of rows to better adapt to the values provided in each row.\nIn the next example, we will see a modern implementation of Graph-RAG with LangChain.\n### LLMGraphTransformer with LangChain\nTo make our graph smarter, we use LangChain to extract entities (movies, actors, etc.) and relationships from text descriptions. `LLMGraphTransformer` is designed to **transform text-based documents into graph documents** using a Language Model.\nLet's start by initializing it:\npython\n\n```\nllm_transformer = LLMGraphTransformer(\n llm=llm,\n)\n```\n\nThe LLM provided is the same one we used for our custom Graph Builder. In this case, we are using the default parameters to give freedom to the model to experiment with nodes, edges, and properties. However, there are a few parameters that are worth knowing to potentially further boost the performances of this algorithm:\n * `**allowed_nodes**`and`**allowed_relationships**`: We haven't specified, so by default, all node and relationship types are allowed.\n * `**strict_mode=True**`: Ensures that only allowed nodes and relationships are included in the output if constraints are specified.\n * `**node_properties=False**`: Disables property extraction for nodes.\n * `**relationship_properties=False**`: Disables property extraction for relationships.\n * `**prompt**`: Pass a`ChatPromptTemplate` to customize the context of the LLM. This is similar to what we have done in our custom LLM.\n\n\nOne caveat of this algorithm is that it's quite slow, especially considering we are not providing a list of nodes and relationships. Therefore, we will only use 100 rows out of the 1000 available in the dataset to speed things up:\npython\n\n```\ndf_sample = df.head(100) # Reduce sample size for faster processing\n```\n\nNext, let's prepare our dataset. We said \"`LLMGraphTransformer` is designed to **transform text-based documents into graph documents\"** , this means that we need to turn our pandas dataframe into text:\npython\n\n```\ndf_sample = movies.head(100)\n\ndocuments = []\nfor _, row in tqdm(df_sample.iterrows(), \n total=len(df_sample), \n desc=\"Creating documents\",\n position=0, \n leave=True):\n try:\n # Format text with proper line breaks\n text = \"\"\n\n for col in df.columns:\n text += f\"{col}: {row[col]}n\"\n\n documents.append(Document(page_content=text))\n\n except KeyError as e:\n tqdm.write(f\"Missing column: {e}\")\n except Exception as e:\n tqdm.write(f\"Error processing row: {e}\")\n```\n\nThis will convert each row into text and add it to a LangChain `Document` object, which is compatible with LangChain's `LLMGraphTransformer` .\nNext, we run the LLM and start the generation:\npython\n\n```\ngraph_documents = await llm_transformer.aconvert_to_graph_documents(documents)\n```\n\nNote that in this case, we are using `await`, and `aconvert_to_graph_documents` instead of `convert_to_graph_documents` to process documents asynchronously, enabling faster execution in large-scale applications.\nNext, sit tight because this will take a few minutes (~30 minutes). Once, the conversion is finished let's print the generated nodes and relationships:\npython\n\n```\nprint(f\"Nodes:{graph_documents[0].nodes}\")\nprint(f\"Relationships:{graph_documents[0].relationships}\")\n```\n\nIn my case, I got the following:\nbash\n\n```\nNodes:[Node(id=\"Boone's cabin\", type='Cabin', properties={}), Node(id='Boone', type='Boone', properties={}), Node(id='Indian maiden', type='Person', properties={}), Node(id='Indian chief', type='Chief', properties={}), Node(id='Florence Lawrence', type='Person', properties={}), Node(id='William Craven', type='Person', properties={}), Node(id='Wallace mccutcheon and ediwin s. porter', type='Director', properties={}), Node(id='Burning arrow', type='Arrow', properties={}), Node(id='Boone', type='Person', properties={}), Node(id='Indian camp', type='Camp', properties={}), Node(id='American', type='Ethnicity', properties={}), Node(id=\"Boone's horse\", type='Horse', properties={}), Node(id='None', type='None', properties={}), Node(id='an indian maiden', type='Maiden', properties={}), Node(id='William craven', type='Cast', properties={}), Node(id='florence lawrence', type='Cast', properties={}), Node(id='Swears vengeance', type='Vengeance', properties={}), Node(id='Daniel Boone', type='Person', properties={}), Node(id='Indians', type='Group', properties={}), Node(id='Daniel boone', type='Title', properties={}), Node(id=\"Daniel Boone's daughter\", type='Person', properties={})]\nRelationships:[Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='Daniel boone', type='Title', properties={}), type='TITLE', properties={}), Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='American', type='Ethnicity', properties={}), type='ORIGIN_ETHNICITY', properties={}), Relationship(source=Node(id='Daniel Boone', type='Person', properties={}), target=Node(id='Wallace mccutcheon and ediwin s. porter', type='Director', properties={}), type='DIRECTED_BY', properties={}), Relationship(source=Node(id='William Craven', type='Person', properties={}), target=Node(id='William craven', type='Cast', properties={}), type='CAST', properties={}), Relationship(source=Node(id='Florence Lawrence', type='Person', properties={}), target=Node(id='florence lawrence', type='Cast', properties={}), type='CAST', properties={}), Relationship(source=Node(id=\"Daniel Boone's daughter\", type='Person', properties={}), target=Node(id='an indian maiden', type='Maiden', properties={}), type='BEFRIENDS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Indian camp', type='Camp', properties={}), type='LEADS_OUT_ON_A_HUNTING_EXPEDITION', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id=\"Boone's cabin\", type='Cabin', properties={}), type='ATTACKS', properties={}), Relationship(source=Node(id='Indian maiden', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='ESCAPES', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Swears vengeance', type='Vengeance', properties={}), type='RETURNS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='HEADS_OUT_ON_THE_TRAIL_TO_THE_INDIAN_CAMP', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id='Boone', type='Boone', properties={}), type='ENCOUNTERS', properties={}), Relationship(source=Node(id='Indian camp', type='Camp', properties={}), target=Node(id='Burning arrow', type='Arrow', properties={}), type='SET_ON_FIRE', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='None', type='None', properties={}), type='GETS_TIED_TO_THE_STAKE_AND_TOURED', properties={}), Relationship(source=Node(id='Indian camp', type='Camp', properties={}), target=Node(id='Burning arrow', type='Arrow', properties={}), type='SETS_ON_FIRE', properties={}), Relationship(source=Node(id='Indians', type='Group', properties={}), target=Node(id=\"Boone's horse\", type='Horse', properties={}), type='ENCOUNTERS', properties={}), Relationship(source=Node(id='Boone', type='Person', properties={}), target=Node(id='Indian chief', type='Chief', properties={}), type='HAS_KNIFE_FIGHT_IN_WHICH_HE_KILLS_THE_INDIAN_CHIEF', properties={})]\n```\n\nNext, it's time to add the generated graph documents to our knowledge graph. We can do that by leveraging the LangChain integration of Neo4j:\npython\n\n```\ngraph = Neo4jGraph(url=uri, username=user, password=password, enhanced_schema=True)\ngraph.add_graph_documents(graph_documents)\n```\n\nLet's store the graph connection in the `graph` variable passing the same URL, username, and password you used at the beginning of this application. Then, let's call the `add_graph_documents` method to add all the graph documents to our database.\nOnce, the process is complete, let's switch to Neo4j Browser one last time, and check our new knowledge graph.\nAs always, run the following query to see the knowledge graph:\nsql\n\n```\nMATCH p=(m:Movie)-[r]-(n)\nRETURN p\nLIMIT 100;\n```\n\nIn my case, the KG looks like this:\n![Knowledge Graph generated by `LLMGraphTransformer - Image by Author`](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/105lmHSpLqeB1DDLuwqQafQ-scaled.jpeg)\nKnowledge Graph generated by **`LLMGraphTransformer** - Image by Author`\nWell, that's a knowledge graph.\nBut we are not done yet. You may remember that our mission is to actually interrogate the knowledge graph to help us find movies. In this article, I will provide a simple Text to Cypher approach which will leverage an LLM to generate a Cypher query, run the Cypher query, and use the retrieved information as context to answer the user query. However, consider that this is just a simple approach, and we will explore advanced retrieval methods in a future article.\nFirst of all, let's refresh the schema of the graph since we will use it to give an understanding of our KG to the LLM:\npython\n\n```\ngraph.refresh_schema()\n```\n\nNow, it's time to set up the QA chain:\npython\n\n```\n# llm = ChatGoogleGenerativeAI(\n# model=\"gemini-1.5-pro\",\n# temperature=0,\n# max_tokens=None,\n# timeout=None,\n# max_retries=2,\n# api_key=api_key\n# )\n\nCYPHER_GENERATION_TEMPLATE = \"\"\"Task:Generate Cypher statement to query a graph database.\nInstructions:\nUse only the provided relationship types and properties in the schema.\nDo not use any other relationship types or properties that are not provided.\nSchema:\n{schema}\nNote: Do not include any explanations or apologies in your responses.\nDo not respond to any questions that might ask anything else than for you to construct a Cypher statement.\nDo not include any text except the generated Cypher statement.\nReturn every node as whole, do not return only the properties.\n\nThe question is:\n{question}\"\"\"\n\nCYPHER_GENERATION_PROMPT = PromptTemplate(\n input_variables=[\"schema\", \"question\"], template=CYPHER_GENERATION_TEMPLATE\n)\n\nchain = GraphCypherQAChain.from_llm(\n llm, \n graph=graph, \n verbose=True, \n allow_dangerous_requests=True, \n return_intermediate_steps=True,\n cypher_prompt=CYPHER_GENERATION_PROMPT\n)\n```\n\nIf you are using Gemini, make sure to switch to a chat model by uncommenting the `llm` variable at the top.\nIn the code above we provide a prompt to the LLM to help it with the generation, and the schema of the graph passing the graph variable.\nFinally, let's ask it something:\npython\n\n```\nchain.run(\"Give me an overview of the movie titled David Copperfield.\")\n```\n\nIn my case, the output is:\nbash\n\n```\nGenerated Cypher:\nMATCH p=(n:Title {id: 'David Copperfield'})-[*1..2]-()\nRETURN p\nFull Context:\n[{'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Florence La Badie'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'RELEASE_YEAR', {'id': '1911'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'GENRE', {'id': 'Drama'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'DIRECTED', {'id': 'Theodore Marston'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'ORIGIN_ETHNICITY', {'id': 'American'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Mignon Anderson'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'William Russell'}]}, {'p': [{'id': 'David Copperfield'}, 'TITLE', {'id': 'David Copperfield'}, 'CAST_IN', {'id': 'Marie Eline'}]}]\nINFO:httpx:HTTP Request: POST http://127.0.0.1:11434/api/generate \"HTTP/1.1 200 OK\"\n\n> Finished chain.\n'David Copperfield is a 1911 American drama film directed by Theodore Marston. The movie stars David Copperfield, Florence La Badie, Mignon Anderson, William Russell, and Marie Eline in various roles. It provides an overview of the life and adventures of David Copperfield through the eyes of his various companions and experiences.'\n```\n\nBy setting `verbose=True` we output the intermediate step: generated Cypher query, and context provided.\n## Conclusion\nThis article served as an introduction to a modern approach to building a knowledge graph. First, we explored a traditional approach and had an overview of the Cypher language, then created a Naïve LLM graph builder to automate the Graph-building process matching the performances achieved during the manual process. Lastly, we went a step further and introduced `LLMGraphTransformer` by LangChain which significantly improved our KG. However, this is just the beginning of our Graph RAG journey specifically our graph builder journey. There are many more modern approaches we need to explore and build from scratch, and we will do that in future articles.\nMoreover, the strategies we talked about above are still not considering the second component of Graph-RAG: the graph retriever. Although improving the actual graph will also improve retrieval performances, we are still not thinking of a retrieval point of view. For example, following the direction we have taken for now, the more labels, nodes, edges, and properties, the better. However, that's actually making it harder for the retriever function to identify the right information to retrieve. Indeed, modern approaches also consider following a tree structure in the knowledge graph, by creating macro-areas and further breaking them down into micro-areas to help the retrieval.\nAs of now, I would like you to test all this code on a different dataset, and get creative with the customization of our Naïve LLM graph-builder. Although `LLMGraphTransformer` seems a very convenient choice to speed up a flexible graph-builder with a minimum amount of code, this method required much more time to build our KG. Moreover, building it from scratch will make sure you are grasping and fully understanding every component behind Graph-RAG.\nGet ready for more Graph-RAG in future articles!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657505387_wlyc0o.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Llm](https://towardsdatascience.com/tag/llm/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Retrieval Augmented Gen](https://towardsdatascience.com/tag/retrieval-augmented-gen/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&title=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&title=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07%2F&text=Building%20a%20Knowledge%20Graph%20From%20Scratch%20Using%20LLMs)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1nSXPIfSQuhm2rKaUk5KuVg.jpeg)](https://towardsdatascience.com/bias-detection-in-machine-learning-models-using-amazon-sagemaker-clarify-d96482692611/)\n## [Bias Detection in Machine Learning Models using Amazon SageMaker Clarify](https://towardsdatascience.com/bias-detection-in-machine-learning-models-using-amazon-sagemaker-clarify-d96482692611/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n1. Introduction\n[Gaurav Shekhar](https://towardsdatascience.com/author/gaurav.shekhar/)\nMarch 11, 20219 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/0G7d49KM9DcxPDysK-scaled.jpg)](https://towardsdatascience.com/introduction-to-dropout-to-regularize-deep-neural-network-8e9d6b1d4386/)\n## [Why Dropout is so effective in Deep Neural Network?](https://towardsdatascience.com/introduction-to-dropout-to-regularize-deep-neural-network-8e9d6b1d4386/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDropout is a simple way to reduce dependencies in the Neural Network\n[Saurav Singla](https://towardsdatascience.com/author/sauravsingla08/)\nAugust 2, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1Rj255KICRRESjWJXppw-Kw.png)](https://towardsdatascience.com/microsofts-new-ai-method-to-predict-how-molecules-move-and-function-93d47e246b5d/)\n## [Microsoft's New AI Method to Predict How Molecules Move and Function](https://towardsdatascience.com/microsofts-new-ai-method-to-predict-how-molecules-move-and-function-93d47e246b5d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTermed the \"Distributional Graphormer\", it could kick-start the next step in the revolution that AI is bringing to the chemical and...\n[Luciano Abriata](https://towardsdatascience.com/author/lucianosphere/)\nJuly 20, 20239 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1mNOrAhYO2N9-DyMW2Kq3HQ.png)](https://towardsdatascience.com/ai-computed-breast-cancer-risk-map-b29195b477a/)\n## [AI Computed Breast Cancer Risk Map](https://towardsdatascience.com/ai-computed-breast-cancer-risk-map-b29195b477a/)\n[](https://towardsdatascience.com/building-a-knowledge-graph-from-scratch-using-llms-f6f677a17f07/)\nA visual and reliable breast cancer diagnosis model using Hierarchical Clustering combined with Boosting and Support Vector Machines\n[jean-baptiste charraud](https://towardsdatascience.com/author/jbcharraud/)\nApril 10, 202118 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/03/0YLuPMkK16ddqsqIh-scaled.jpg)](https://towardsdatascience.com/how-machine-learning-will-become-a-game-changer-for-the-location-data-industry-10dde8f41f3e/)\n## [How Machine Learning Will Become a Game Changer for the Location Data Industry](https://towardsdatascience.com/how-machine-learning-will-become-a-game-changer-for-the-location-data-industry-10dde8f41f3e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLocation data can provide unique insights but comes with costs and privacy issues. ML can overcome these drawbacks and improve location...\n[Benjamin Thürer](https://towardsdatascience.com/author/benjamin.thuerer/)\nMarch 9, 202310 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1OJUmmet0I_Y6VMCGNRQiqw.png)](https://towardsdatascience.com/explaining-openai-soras-spacetime-patches-the-key-ingredient-e14e0703ec5b/)\n## [Explaining OpenAI Sora's Spacetime Patches: The Key Ingredient](https://towardsdatascience.com/explaining-openai-soras-spacetime-patches-the-key-ingredient-e14e0703ec5b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnder The Hood Of The Generative AI For Video By OpenAI\n[Vincent Koc](https://towardsdatascience.com/author/vincentkoc/)\nFebruary 16, 20246 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/1-gFcUuT5YRbEjqyF-F0ElQ-scaled.jpeg)](https://towardsdatascience.com/how-to-become-a-pandas-jedi-bad82667b34c/)\n## [How to Become a Pandas Jedi](https://towardsdatascience.com/how-to-become-a-pandas-jedi-bad82667b34c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA practical guide to boost your data analysis skills\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nFebruary 7, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/12cMo4bAgsAHZrxR-M7b_WA.jpeg)](https://towardsdatascience.com/stop-using-csvs-for-storage-this-file-format-is-150-times-faster-158bd322074e/)\n## [Stop Using CSVs for Storage - This File Format Is 150 Times Faster](https://towardsdatascience.com/stop-using-csvs-for-storage-this-file-format-is-150-times-faster-158bd322074e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nCSV's are costing you time, disk space, and money. It's time to end it.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nSeptember 6, 20214 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1N720BDToAzZBkuKcby0UBw.png)](https://towardsdatascience.com/intuitively-understand-roc-and-implement-it-in-r-and-python-b6aa1e8508d/)\n## [Intuitively understand ROC and implement it in R and Python](https://towardsdatascience.com/intuitively-understand-roc-and-implement-it-in-r-and-python-b6aa1e8508d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA case-study based introduction to ROC, how to set the threshold, its relation to sensitivity/specificity/precision/recall/accuracy, and...\n[Ahmar Shah, PhD (Oxford)](https://towardsdatascience.com/author/ahmarshah/)\nDecember 9, 202012 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/0852990cc19f/pages/", "body_markdown_path": "attachments/0852990cc19f/pages/archived-body.md", "local_html": "attachments/0852990cc19f/pages/archived-body.html", "body_source": "prior_capture" }