Generators in Python: Fundamentals for Data Scientists

Understand the basics with a concrete example!

Image for post
Image for post
Photo by Oskar Yildiz on Unsplash
  • Both yield and return statements return a value from a function. While the return statement ends the function completely, yield statement suspends the function by keeping all its state in the memory for later use.
  • When the generator functions yields, the function is not terminated. Instead, it pauses the function and gives control over to the caller.
  • After fully iterated, generators terminate and raise stopIteration exception.
# Simple Generator
def my_simple_generator():
k = 0
yield k
k = 1
yield k
k = 2
yield k
my_numbers = my_simple_generator()print(next(my_numbers))
print(next(my_numbers))
print(next(my_numbers))
# Below call raises StopIteration as generator is fully iterated
# print(next(my_numbers))
# Defining Generators with Loop
def my_generator_with_loop(my_str):
length = len(my_str)
for k in range(length):
yield my_str[k]
my_text = my_generator_with_loop("Coding")for char in my_text:
print(char)
# Defining Generators with Expressions
my_generator_expression = (number**2 for number in range(4))
print (sum(my_generator_expression))
# Defining Generator Pipeline
my_generator_01 = (number**2 for number in range(40))
my_generator_02 = (number-5 for number in my_generator_01)
print(sum(my_generator_02))
Image for post
Image for post
Photo by alex lauzon on Unsplash

Defining Generator Functions

# Simple Generator
def my_simple_generator():
k = 0
yield k
k = 1
yield k
k = 2
yield k
my_numbers = my_simple_generator()print(next(my_numbers))
print(next(my_numbers))
print(next(my_numbers))
# Below call raises StopIteration as generator is fully iterated
# print(next(my_numbers))
#Output
0
1
2
Image for post
Image for post
Photo by Norja V on Unsplash

Defining Generators with a Loop

To simplify the generator definition we can define and iterate the generators with the help of for loops.

# Defining Generators with Loop
def my_generator_with_loop(my_str):
length = len(my_str)
for k in range(length):
yield my_str[k]
my_text = my_generator_with_loop("Coding")for char in my_text:
print(char)
#Output
C
o
d
i
n
g

Defining Generators With Generator Expressions

We can further simplify the definition of generators with generator expressions. The syntax for creating a generator expression is very similar to the one used for list comprehension. Round brackets are used to create generator expressions, instead of square brackets used in the list comprehension.

# Defining Generators with Expressions
my_generator_expression = (number**2 for number in range(4))
print (sum(my_generator_expression))
#Output
14

Defining Generator Pipelines

To create complex pipelines, multiple generators can be stacked easily as shown in the below example.

# Defining Generator Pipeline
my_generator_01 = (number**2 for number in range(40))
my_generator_02 = (number-5 for number in my_generator_01)
print(sum(my_generator_02))#Output
20340
Image for post
Image for post
Photo by alex lauzon on Unsplash

Key Takeaways

  • Generators are memory-friendly as they return and store the portion of data only when it is demanded.
  • Generators simplify the iterator definition process
  • We can define generators with generators expressions or generator functions.
  • We can develop memory-efficient data pipelines by using multiple generators.

Conclusion

In this post, I explained the basics of generators in Python.

Machine Learning and Data Science Enthusiasts, Automotive Engineer, Mechanical Engineer, https://www.linkedin.com/in/erdem-isbilen/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store