Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying IPython Interactive Computing and Visualization Cookbook
  • Table Of Contents Toc
  • Feedback & Rating feedback
IPython Interactive Computing and Visualization Cookbook

IPython Interactive Computing and Visualization Cookbook

By : Cyrille Rossant
4.5 (13)
close
close
IPython Interactive Computing and Visualization Cookbook

IPython Interactive Computing and Visualization Cookbook

4.5 (13)
By: Cyrille Rossant

Overview of this book

Intended to anyone interested in numerical computing and data science: students, researchers, teachers, engineers, analysts, hobbyists... Basic knowledge of Python/NumPy is recommended. Some skills in mathematics will help you understand the theory behind the computational methods.
Table of Contents (22 chapters)
close
close
IPython Interactive Computing and Visualization Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Processing huge NumPy arrays with memory mapping


Sometimes, we need to deal with NumPy arrays that are too big to fit in the system memory. A common solution is to use memory mapping and implement out-of-core computations. The array is stored in a file on the hard drive, and we create a memory-mapped object to this file that can be used as a regular NumPy array. Accessing a portion of the array results in the corresponding data being automatically fetched from the hard drive. Therefore, we only consume what we use.

How to do it...

  1. Let's create a memory-mapped array:

    In [1]: import numpy as np
    In [2]: nrows, ncols = 1000000, 100
    In [3]: f = np.memmap('memmapped.dat', dtype=np.float32, 
                          mode='w+', shape=(nrows, ncols))
  2. Let's feed the array with random values, one column at a time because our system's memory is limited!

    In [4]: for i in range(ncols):
                f[:,i] = np.random.rand(nrows)

    We save the last column of the array:

    In [5]: x = f[:,-1]
  3. Now, we flush memory changes...

Limited Time Offer

$10p/m for 3 months

Get online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech and supported with AI assistants
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon

Create a Note

Modal Close icon
You need to login to use this feature.

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Edit Note

Modal Close icon
Write a note (max 255 characters)
Cancel
Update Note