{"id":5634,"date":"2024-11-29T09:14:20","date_gmt":"2024-11-29T09:14:20","guid":{"rendered":"https:\/\/ubiq.co\/tech-blog\/?p=5634"},"modified":"2024-11-29T09:26:55","modified_gmt":"2024-11-29T09:26:55","slug":"how-to-read-large-file-in-python","status":"publish","type":"post","link":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/","title":{"rendered":"How to Read Large File in Python"},"content":{"rendered":"\n<p>These days almost every website and application needs to work with files. With growing number of users, these files are fast growing in size. Often Python software developers need to be able to read large files while building their applications. There are several ways to read files in Python. But if you are not careful, it may load the entire file in your system&#8217;s memory and bring it down. In this article, we will learn how to read large file in Python.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#The_Problem\" >The Problem<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#How_to_Read_Large_File_in_Python\" >How to Read Large File in Python<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#What_is_Iterator\" >What is Iterator<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#1_Using_Iterator_of_File_Object\" >1. Using Iterator of File Object<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#2_Using_fileinput\" >2. Using fileinput<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#3_Using_read\" >3. Using read()<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#4_Using_Python_Pandas\" >4. Using Python Pandas<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#5_Using_Dask\" >5. Using Dask<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#Conclusion\" >Conclusion<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#FAQ\" >FAQ<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_Problem\"><\/span>The Problem<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Typically, when a file is read by Python or any programming language, it is completely loaded into memory. If the file size is too large, say in GB, then this may overwhelm the memory and crash your system or server, wherever it is being processed. That is why we need to ensure that only the bare minimum part of your file is actually loaded into your system&#8217;s memory. In this article, we will learn how to do this.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_Read_Large_File_in_Python\"><\/span>How to Read Large File in Python<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Generally, files are read in Python using readlines() method, which loads the entire file in memory. It returns a list where each element is a line of the file. This list is stored in memory. If your file is large, then by the time your entire file is read, it will occupy a large amount of space in memory. Also, every time it reads a line, it is appended to the list. So it is also time-consuming. Here is a sample code to read the file \/home\/test.txt using readlines().<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">f = open('\/home\/test.txt','r'):<br>l= f.readlines()<br>f.close()<\/pre>\n\n\n\n<p>The above code will return a list consisting of target file&#8217;s lines. To fix this problem, we need to use a file iterator object. Before we proceed, let us learn what an iterator is.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_is_Iterator\"><\/span>What is Iterator<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>An iterator is simply an object with countable number of values. You can iterate upon these values sequentially, using a loop. It will keep returning one value at a time, till it reaches end of values. An iterator is like a pointer to the values so it does not load all values in the memory. So it does not require much storage and is memory efficient. Every file is an iterable in Python. So if you want to read a large file, it is better to use an iterator to a file object.<\/p>\n\n\n\n<p>Let us look at some of the simple ways to easily read a large file in Python using iterators.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"1_Using_Iterator_of_File_Object\"><\/span>1. Using Iterator of File Object<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this case, we use an iterator of a file object. For this purpose, we use open() function, that automatically opens the file as well as returns a file object for it. Thereafter, we use an iterator to the file object to iterate over the file and print each line. You can customize the following code to do other things, instead of printing the line.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">with open(\"\/home\/test.txt\") as file:<br>    for line in file:<br>        print(line)<\/pre>\n\n\n\n<p>Once the file is completely read, it will be automatically closed, thanks to open() function. All this happens internally in this case, and is not explicitly visible.<\/p>\n\n\n\n<p>This is one of the fastest ways to read a large file in Python. In fact, you can use the same method for all files, big or small.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"2_Using_fileinput\"><\/span>2. Using fileinput<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>fileinput is a Python module that helps you to easily loop over list of files, or standard input. It provides an input() function for this purpose. It returns an iterator to the file object. Here is a sample code to illustrate its use.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">import fileinput<br><br>path = '\/home\/test.txt'<br>for line in fileinput.input([path])<br>    print(line)<\/pre>\n\n\n\n<p>Please remember that the input() function requires a list of file paths. So you need to input the list of file path strings and not just direct strings of file paths.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"3_Using_read\"><\/span>3. Using read()<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Both the above methods rely on sequentially reading individual lines from a file. What if your file does not contain separate lines but is a continuous series of characters. What if your file does not contain newline character? In this case, you will need to read the file content as chunks.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">path = '\/home\/test.txt'<br>with open(path) as f:<br>  while True:<br>    c = f.read(1024)<br>    if not c:<br>      break<br>    print(c,end='')<\/pre>\n\n\n\n<p>In the above code, we have used a chunk size of 1024 bytes. You can adjust it as per your requirements. We open the file using open() function. Then we run a while loop. In each iteration, we use read() function to read a chunk of file&#8217;s data. This goes on as long as there is unread data in your file. When all its data has been read, the code will execute the bread statement and end loop. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"4_Using_Python_Pandas\"><\/span>4. Using Python Pandas<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><a href=\"https:\/\/pandas.pydata.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">Pandas<\/a> is a popular Python library that allows you to easily read and process data, as dataframes, which consist of rows and columns. It provides read_csv() function to easily read text as well as csv files, big or small. Here is a sample code to read a large file using Python pandas.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">import pandas as pd<br><br>path = '\/home\/test.txt'<br>chunk_size = 1000 # no. of rows per chunk<br><br>for chunk in pd.read_csv(path, chunksize=chunk_size):<br>    for index, row in chunk.iterrows():<br>        print(row)<\/pre>\n\n\n\n<p>In Pandas&gt;=1.2<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">import pandas as pd<br><br>path = '\/home\/test.txt'<br>chunk_size = 1000 # no. of rows per chunk<br><br>with pd.read_csv(path, chunksize=chunk_size) as reader:<br>    for chunk in reader:<br>        process(chunk)<\/pre>\n\n\n\n<p>In the above code, we read 1000 lines of data in each iteration. Each chunk is a dataframe. You can change it as per your requirement.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"5_Using_Dask\"><\/span>5. Using Dask <span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><a href=\"https:\/\/www.dask.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">Dask<\/a> is a Python library for parallel processing. Dask Dataframe allows you to process Pandas dataframe in a parallel manner. This is super useful for really large files (>100Gb).<\/p>\n\n\n\n<p>If you are using Dask Dataframe, then it also provides read_csv() function that allows you to read large files, like Pandas. Here is a code to demonstrate it. Unlike pandas, dask handles chunk size internally on its own. So you do not need to explicitly specify it.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">import dask.dataframe as dd<br><br>path = '\/home\/test.txt'<br>df = dd.read_csv(path)<br><br>for idx, row in df.iterrows():<br>    print(row)<\/pre>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In this article, we have learnt several ways to read large file in Python. You can use any of these methods. The key point is to use iterator to file object so that the entire file is not loaded into memory all at once. A file iterator object is only a pointer to the file and allows you to read the file in manageable chunks without overwhelming your system&#8217;s memory. All the methods listed above use the same concept, of an iterator to a file object. You can use any of them as per your requirement.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"FAQ\"><\/span>FAQ<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><strong>1. Can I use the above solution for any file type?<\/strong><\/p>\n\n\n\n<p>Yes. You can use the above solution for any file type. The first solutions read the files line by line so they work especially well for text and csv files. Solution #3 works for pretty much all files whether their content is organized as lines or not.<\/p>\n\n\n\n<p><strong>2. Is there a file limit for which these solutions work?<\/strong><\/p>\n\n\n\n<p>No. Each solution is scalable and works for large files. But if you have really large files (>100 Gb), then you may want to try solution #5 since it employs parallel processing.<\/p>\n\n\n\n<p><strong>3. Can they be used in another Python script?<\/strong><\/p>\n\n\n\n<p>Yes. You can easily include them in your Python script, website or application.<\/p>\n\n\n\n<p>Also read:<\/p>\n\n\n\n<p><a href=\"https:\/\/ubiq.co\/tech-blog\/how-to-execute-system-command-in-python\/\">How to Execute System Command in Python<\/a><br><a href=\"https:\/\/ubiq.co\/tech-blog\/how-to-check-if-file-exists-in-python\/\">How to Check if File Exists in Python<\/a><br><a href=\"https:\/\/ubiq.co\/tech-blog\/how-to-merge-two-dictionaries-in-python\/\">How to Merge Two Dictionaries in Python<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.<\/p>\n","protected":false},"author":1,"featured_media":5657,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[301],"tags":[339],"class_list":["post-5634","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-large-file"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to Read Large File in Python - Ubiq BI<\/title>\n<meta name=\"description\" content=\"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Read Large File in Python - Ubiq BI\" \/>\n<meta property=\"og:description\" content=\"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Ubiq BI\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ubiqbi\" \/>\n<meta property=\"article:published_time\" content=\"2024-11-29T09:14:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-11-29T09:26:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"301\" \/>\n\t<meta property=\"og:image:height\" content=\"201\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Sreeram Sreenivasan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@UbiqBI\" \/>\n<meta name=\"twitter:site\" content=\"@UbiqBI\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sreeram Sreenivasan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/\"},\"author\":{\"name\":\"Sreeram Sreenivasan\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/#\\\/schema\\\/person\\\/db98d49a766a3a111d8510935ab90abc\"},\"headline\":\"How to Read Large File in Python\",\"datePublished\":\"2024-11-29T09:14:20+00:00\",\"dateModified\":\"2024-11-29T09:26:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/\"},\"wordCount\":1149,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/ubiq.co\\\/tech-blog\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1\",\"keywords\":[\"large file\"],\"articleSection\":[\"Python\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/\",\"url\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/\",\"name\":\"How to Read Large File in Python - Ubiq BI\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/ubiq.co\\\/tech-blog\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1\",\"datePublished\":\"2024-11-29T09:14:20+00:00\",\"dateModified\":\"2024-11-29T09:26:55+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/#\\\/schema\\\/person\\\/db98d49a766a3a111d8510935ab90abc\"},\"description\":\"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/ubiq.co\\\/tech-blog\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/ubiq.co\\\/tech-blog\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1\",\"width\":301,\"height\":201,\"caption\":\"read large files in python\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/how-to-read-large-file-in-python\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Read Large File in Python\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/#website\",\"url\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/\",\"name\":\"Ubiq BI\",\"description\":\"Build dashboards &amp; reports in minutes\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/#\\\/schema\\\/person\\\/db98d49a766a3a111d8510935ab90abc\",\"name\":\"Sreeram Sreenivasan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"caption\":\"Sreeram Sreenivasan\"},\"description\":\"Sreeram Sreenivasan is the Founder of Ubiq. He has helped many Fortune 500 companies in the areas of BI &amp; software development.\",\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/in\\\/sreeram-sreenivasan\\\/\"],\"url\":\"https:\\\/\\\/ubiq.co\\\/tech-blog\\\/author\\\/wordpress\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Read Large File in Python - Ubiq BI","description":"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/","og_locale":"en_US","og_type":"article","og_title":"How to Read Large File in Python - Ubiq BI","og_description":"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.","og_url":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/","og_site_name":"Ubiq BI","article_publisher":"https:\/\/www.facebook.com\/ubiqbi","article_published_time":"2024-11-29T09:14:20+00:00","article_modified_time":"2024-11-29T09:26:55+00:00","og_image":[{"width":301,"height":201,"url":"https:\/\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg","type":"image\/jpeg"}],"author":"Sreeram Sreenivasan","twitter_card":"summary_large_image","twitter_creator":"@UbiqBI","twitter_site":"@UbiqBI","twitter_misc":{"Written by":"Sreeram Sreenivasan","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#article","isPartOf":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/"},"author":{"name":"Sreeram Sreenivasan","@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc"},"headline":"How to Read Large File in Python","datePublished":"2024-11-29T09:14:20+00:00","dateModified":"2024-11-29T09:26:55+00:00","mainEntityOfPage":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/"},"wordCount":1149,"commentCount":0,"image":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1","keywords":["large file"],"articleSection":["Python"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/","url":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/","name":"How to Read Large File in Python - Ubiq BI","isPartOf":{"@id":"https:\/\/ubiq.co\/tech-blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#primaryimage"},"image":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1","datePublished":"2024-11-29T09:14:20+00:00","dateModified":"2024-11-29T09:26:55+00:00","author":{"@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc"},"description":"Sometimes software developers need to process large files in Python script or application. Here is how to read large file in Python.","breadcrumb":{"@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#primaryimage","url":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1","contentUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1","width":301,"height":201,"caption":"read large files in python"},{"@type":"BreadcrumbList","@id":"https:\/\/ubiq.co\/tech-blog\/how-to-read-large-file-in-python\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ubiq.co\/tech-blog\/"},{"@type":"ListItem","position":2,"name":"How to Read Large File in Python"}]},{"@type":"WebSite","@id":"https:\/\/ubiq.co\/tech-blog\/#website","url":"https:\/\/ubiq.co\/tech-blog\/","name":"Ubiq BI","description":"Build dashboards &amp; reports in minutes","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ubiq.co\/tech-blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc","name":"Sreeram Sreenivasan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","caption":"Sreeram Sreenivasan"},"description":"Sreeram Sreenivasan is the Founder of Ubiq. He has helped many Fortune 500 companies in the areas of BI &amp; software development.","sameAs":["https:\/\/www.linkedin.com\/in\/sreeram-sreenivasan\/"],"url":"https:\/\/ubiq.co\/tech-blog\/author\/wordpress\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2024\/11\/read-large-files-in-python.jpg?fit=301%2C201&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/pbGGTT-1sS","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/5634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/comments?post=5634"}],"version-history":[{"count":36,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/5634\/revisions"}],"predecessor-version":[{"id":5676,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/5634\/revisions\/5676"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/media\/5657"}],"wp:attachment":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/media?parent=5634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/categories?post=5634"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/tags?post=5634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}