Cannot convert a symbolic Keras input/output to a numpy array TypeError when using sampled_softmax in tensorflow 2.4

I'm trying to train a word embedding classifier using TF2.4 with Keras and using the tf.nn.sampled_softmax_loss. However, when calling the fit method of the model, "Cannot convert a symbolic Keras input/output to a numpy array" TypeError occurs. Please help me to fix the error or with an alternative approach to do candidate sampling.

import tensorflow as tf
import numpy as np
TextVectorization = tf.keras.layers.experimental.preprocessing.TextVectorization
class SampledSoftmaxLoss: #(tf.keras.losses.Loss): def __init__(self, model, n_classes): self.model = model output_layer = model.layers[-1] self.input = output_layer.input self.weights = output_layer.weights self.n_classes = n_classes def loss(self, y_true, y_pred, **kwargs): labels = tf.argmax(y_true, axis=1) labels = tf.expand_dims(labels, -1) loss = tf.nn.sampled_softmax_loss( weights=self.weights[0], biases=self.weights[1], labels=labels, inputs=self.input, num_sampled = 3, num_classes = self.n_classes ) return loss
max_features = 50 # Maximum vocab size.
max_len = 10 # Sequence length to pad the outputs to.
embedding_dims = 5
input_data = np.array([ "Python Machine Learning", "Data Science from Scratch: First Principles with Python", "Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques for Building Intelligent Systems", "Introduction to Machine Learning with Python: A Guide for Data Scientists", "Vital Introduction to Machine Learning with Python: Best Practices to Improve and Optimize Machine Learning Systems and Algorithms", "Machine Learning in Python: Essential Techniques for Predictive Analysis", "Python Data Science Handbook: Essential Tools for Working with Data", "Introducing Data Science: Big Data, Machine Learning, and more, using Python tools", "Real-World Machine Learning"])
labels_one_hot = []
for i in range(len(input_data)): labels = np.random.randint(0, 6, max_features) labels[labels>1] = 1 labels_one_hot.append(labels)
labels_one_hot = np.array(labels_one_hot)
# Create the text categorical encoding layer.
vectorize_layer = TextVectorization( max_tokens=max_features, output_mode='int', output_sequence_length=max_len)
vectorize_layer.adapt(text_dataset)
inp = tf.keras.Input(shape=(1,), dtype=tf.string)
idxs = vectorize_layer(inp)
embed = tf.keras.layers.Embedding(max_features + 1, embedding_dims,input_length=max_len)(idxs)
flat = tf.keras.layers.Flatten()(embed)
out = tf.keras.layers.Dense(labels_one_hot.shape[1])(flat)
model = tf.keras.models.Model(inp, out)
softmax = SampledSoftmaxLoss(model, labels_one_hot.shape[1])
model.compile('adam', loss=softmax.loss)
model.summary()
model.fit(input_data, labels_one_hot)
model.predict(input_data)

This is the log:

INFO:tensorflow:Assets written to: model/assets
Model: "model_5"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_84 (InputLayer) [(None, 1)] 0
_________________________________________________________________
text_vectorization_83 (TextV (None, 10) 0
_________________________________________________________________
embedding_83 (Embedding) (None, 10, 5) 255
_________________________________________________________________
flatten_25 (Flatten) (None, 50) 0
_________________________________________________________________
dense_69 (Dense) (None, 50) 2550
=================================================================
Total params: 2,805
Trainable params: 2,805
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-87-c35b8d1321a1> in <module>() 106 model.compile('adam', loss=softmax.loss) 107 model.summary()
--> 108 model.fit(input_data, labels_one_hot) 109 # Now, the model can map strings to integers, and you can add an embedding 110 # layer to map these integers to learned embeddings.
9 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs) 975 except Exception as e: # pylint:disable=broad-except 976 if hasattr(e, "ag_error_metadata"):
--> 977 raise e.ag_error_metadata.to_exception(e) 978 else: 979 raise
TypeError: in user code: /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:805 train_function * return step_function(self, iterator) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:795 step_function ** outputs = model.distribute_strategy.run(run_step, args=(data,)) /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1259 run return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2730 call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:3417 _call_for_each_replica return fn(*args, **kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:788 run_step ** outputs = model.train_step(data) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:756 train_step y, y_pred, sample_weight, regularization_losses=self.losses) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:238 __call__ total_loss_metric_value, sample_weight=batch_dim) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/metrics_utils.py:90 decorated update_op = update_state_fn(*args, **kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/metrics.py:177 update_state_fn return ag_update_state(*args, **kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/metrics.py:364 update_state ** sample_weight, values) /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/weights_broadcast_ops.py:155 broadcast_weights values = ops.convert_to_tensor(values, name="values") /usr/local/lib/python3.6/dist-packages/tensorflow/python/profiler/trace.py:163 wrapped return func(*args, **kwargs) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:1540 convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py:339 _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py:265 constant allow_broadcast=True) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py:283 _constant_impl allow_broadcast=allow_broadcast)) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_util.py:435 make_tensor_proto values = np.asarray(values) /usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py:83 asarray return array(a, dtype, copy=False, order=order) /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/keras_tensor.py:274 __array__ 'Cannot convert a symbolic Keras input/output to a numpy array. ' TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.

4 Answers

The error is caused by your custom loss function. You should disable TF eager execution mode.

from tensorflow.python.framework.ops import disable_eager_execution

disable_eager_execution()

Also, in model.compile(...) use experimental_run_tf_function=False

The following solution has saved me a bit of headache, but I am not certain why it works. I am new to DL and I am just trying to be helpful.

Try:

del model

Before:

model = tf.keras.models.Model(inp, out)

I understand I could be wrong feel free to educate me, I am willing to learn :) .

I added these two commands and solved the problem:

import tensorflow._api.v2.compat.v1 as tf
tf.disable_v2_behavior()

one of the main reasons for this error is that you may have installed Keras before TensorFlow. Restart the kernel and you will be OK, if this is the case