Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
429 views
in Technique[技术] by (71.8m points)

python - How to make a generator callable?

I'm trying to create a dataset from a CSV file with 784-bit long rows. Here's my code:

import tensorflow as tf

f = open("test.csv", "r")
csvreader = csv.reader(f)
gen = (row for row in csvreader)
ds = tf.data.Dataset()
ds.from_generator(gen, [tf.uint8]*28**2)

I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-4b244ea66c1d> in <module>()
     12 gen = (row for row in csvreader_pat_trn)
     13 ds = tf.data.Dataset()
---> 14 ds.from_generator(gen, [tf.uint8]*28**2)

~/Documents/Programming/ANN/labs/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py in from_generator(generator, output_types, output_shapes)
    317     """
    318     if not callable(generator):
--> 319       raise TypeError("`generator` must be callable.")
    320     if output_shapes is None:
    321       output_shapes = nest.map_structure(

TypeError: `generator` must be callable.

The docs said that I should have a generator passed to from_generator(), so that's what I did, gen is a generator. But now it's complaining that my generator isn't callable. How can I make the generator callable so I can get this to work?

EDIT: I'd like to add that I'm using python 3.6.4. Is this the reason for the error?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

The generator argument (perhaps confusingly) should not actually be a generator, but a callable returning an iterable (for example, a generator function). Probably the easiest option here is to use a lambda. Also, a couple of errors: 1) tf.data.Dataset.from_generator is meant to be called as a class factory method, not from an instance 2) the function (like a few other in TensorFlow) is weirdly picky about parameters, and it wants you to give the sequence of dtypes and each data row as tuples (instead of the lists returned by the CSV reader), you can use for example map for that:

import csv
import tensorflow as tf

with open("test.csv", "r") as f:
    csvreader = csv.reader(f)
    ds = tf.data.Dataset.from_generator(lambda: map(tuple, csvreader),
                                        (tf.uint8,) * (28 ** 2))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...