nn¶

Adaptive Pool2d Operator The adaptive_pool2d operation calculates the output based on the input, pool_size, pool_type parameters. Input(X) and output(Out) are in NCHW format, where N is batch size, C is the number of channels, H is the height of the feature, and W is the width of the feature. Parameters(pool_size) should contain two elements which represent height and width, respectively. Also the H and W dimensions of output(Out) is same as Parameter(pool_size).

For average adaptive pool2d:

Parameters

• input (Variable) – The input tensor of pooling operator. The format of input tensor is NCHW, where N is batch size, C is the number of channels, H is the height of the feature, and W is the width of the feature.

• pool_size (int|list|tuple) – The pool kernel size. If pool kernel size is a tuple or list, it must contain two integers, (pool_size_Height, pool_size_Width).

• pool_type – (string), pooling type, can be “max” for max-pooling and “avg” for average-pooling

• require_index (bool) – If true, the index of max pooling point will be returned along with outputs. It cannot be set in average pooling type.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The pooling result.

Return type

Variable

Raises

• ValueError – ‘pool_type’ is not ‘max’ nor ‘avg’.

• ValueError – invalid setting ‘require_index’ true when ‘pool_type’ is ‘avg’.

• ValueError – ‘pool_size’ should be a list or tuple with length as 2.

Examples

# suppose input data in shape of [N, C, H, W], `pool_size` is [m, n],
# output shape is [N, C, m, n], adaptive pool divide H and W dimentions
# of input data into m * n grids averagely and performs poolings in each
# grid to get output.
# adaptive average pool performs calculations as follow:
#
#     for i in range(m):
#         for j in range(n):
#             hstart = floor(i * H / m)
#             hend = ceil((i + 1) * H / m)
#             wstart = floor(i * W / n)
#             wend = ceil((i + 1) * W / n)
#             output[:, :, i, j] = avg(input[:, :, hstart: hend, wstart: wend])
#
import paddle.fluid as fluid
data = fluid.layers.data(
    name='data', shape=[3, 32, 32], dtype='float32')
pool_out = fluid.layers.adaptive_pool2d(
                  input=data,
                  pool_size=[3, 3],
                  pool_type='avg')

Adaptive Pool3d Operator The adaptive_pool3d operation calculates the output based on the input, pool_size, pool_type parameters. Input(X) and output(Out) are in NCDHW format, where N is batch size, C is the number of channels, D is the depth of the feature, H is the height of the feature, and W is the width of the feature. Parameters(pool_size) should contain three elements which represent height and width, respectively. Also the D, H and W dimensions of output(Out) is same as Parameter(pool_size).

For average adaptive pool3d:

Parameters

• input (Variable) – The input tensor of pooling operator. The format of input tensor is NCDHW, where N is batch size, C is the number of channels, D is the depth of the feature, H is the height of the feature, and W is the width of the feature.

• pool_size (int|list|tuple) – The pool kernel size. If pool kernel size is a tuple or list, it must contain three integers, (Depth, Height, Width).

• pool_type – (string) Pooling type, can be “max” for max-pooling and “avg” for average-pooling

• require_index (bool) – If true, the index of max pooling point will be returned along with outputs. It cannot be set in average pooling type.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The pooling result.

Return type

Variable

Raises

• ValueError – ‘pool_type’ is not ‘max’ nor ‘avg’.

• ValueError – invalid setting ‘require_index’ true when ‘pool_type’ is ‘avg’.

• ValueError – ‘pool_size’ should be a list or tuple with length as 2.

Examples

# suppose input data in shape of [N, C, D, H, W], `pool_size` is [l, m, n],
# output shape is [N, C, l, m, n], adaptive pool divide D, H and W dimentions
# of input data into l * m * n grids averagely and performs poolings in each
# grid to get output.
# adaptive average pool performs calculations as follow:
#
#     for i in range(l):
#         for j in range(m):
#             for k in range(n):
#                 dstart = floor(i * D / l)
#                 dend = ceil((i + 1) * D / l)
#                 hstart = floor(j * H / m)
#                 hend = ceil((j + 1) * H / m)
#                 wstart = floor(k * W / n)
#                 wend = ceil((k + 1) * W / n)
#                 output[:, :, i, j, k] =
#                     avg(input[:, :, dstart:dend, hstart: hend, wstart: wend])
#

import paddle.fluid as fluid

data = fluid.layers.data(
    name='data', shape=[3, 32, 32, 32], dtype='float32')
pool_out = fluid.layers.adaptive_pool3d(
                  input=data,
                  pool_size=[3, 3, 3],
                  pool_type='avg')

Add Position Encoding Layer

This layer accepts an input 3D-Tensor of shape [N x M x P], and returns an output Tensor of shape [N x M x P] with positional encoding value.

Refer to Attention Is All You Need .

Where:

• \(PE(pos, 2i)\) : the increment for the number at even position

• \(PE(pos, 2i + 1)\) : the increment for the number at odd position

Parameters

• input (Variable) – 3-D input tensor with shape [N x M x P]

• alpha (float) – multiple of Input Tensor

• beta (float) – multiple of Positional Encoding Tensor

• name (string) – the name of position encoding layer

Returns

A 3-D Tensor of shape [N x M x P] with positional encoding.

Return type

Variable

Examples

import paddle.fluid as fluid

tensor = fluid.layers.data(
    name='tensor',
    shape=[32, 64, 512],
    dtype='float32',
    append_batch_size=False)
position_tensor = fluid.layers.add_position_encoding(
    input=tensor, alpha=1.0, beta=1.0)

Applies a separate affine transformation to each channel of the input. Useful for replacing spatial batch norm with its equivalent fixed transformation. The input also can be 2D tensor and applies a affine transformation in second dimension.

Parameters

• x (Variable) – Feature map input can be a 4D tensor with order NCHW or NHWC. It also can be a 2D tensor and the affine transformation is applied in the second dimension.

• scale (Variable) – 1D input of shape (C), the c-th element is the scale factor of the affine transformation for the c-th channel of the input.

• bias (Variable) – 1D input of shape (C), the c-th element is the bias of the affine transformation for the c-th channel of the input.

• data_layout (string, default NCHW) – NCHW or NHWC. If input is 2D tensor, you can ignore data_layout.

• name (str, default None) – The name of this layer.

• act (str, default None) – Activation to be applied to the output of this layer.

Returns

A tensor of the same shape and data layout with x.

Return type

out (Variable)

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 32, 32],
                         dtype='float32')
input_scale = fluid.layers.create_parameter(shape=[3],
                         dtype="float32")
input_bias = fluid.layers.create_parameter(shape=[3],
                         dtype="float32")
out = fluid.layers.affine_channel(data,scale=input_scale,
                         bias=input_bias)

It generates a grid of (x,y) coordinates using the parameters of the affine transformation that correspond to a set of points where the input feature map should be sampled to produce the transformed output feature map.

* Case 1:

  Given:

      theta = [[[x_11, x_12, x_13]
                [x_14, x_15, x_16]]
               [[x_21, x_22, x_23]
                [x_24, x_25, x_26]]]

      out_shape = [2, 3, 5, 5]

  Step 1:

      Generate normalized coordinates according to out_shape.
      The values of the normalized coordinates are in the interval between -1 and 1.
      The shape of the normalized coordinates is [2, H, W] as below:

      C = [[[-1.  -1.  -1.  -1.  -1. ]
            [-0.5 -0.5 -0.5 -0.5 -0.5]
            [ 0.   0.   0.   0.   0. ]
            [ 0.5  0.5  0.5  0.5  0.5]
            [ 1.   1.   1.   1.   1. ]]
           [[-1.  -0.5  0.   0.5  1. ]
            [-1.  -0.5  0.   0.5  1. ]
            [-1.  -0.5  0.   0.5  1. ]
            [-1.  -0.5  0.   0.5  1. ]
            [-1.  -0.5  0.   0.5  1. ]]]
      C[0] is the coordinates in height axis and  C[1] is the coordinates in width axis.

  Step2:

      Tanspose and reshape C to shape [H * W, 2] and append ones to last dimension. The we get:
      C_ = [[-1.  -1.   1. ]
            [-0.5 -1.   1. ]
            [ 0.  -1.   1. ]
            [ 0.5 -1.   1. ]
            [ 1.  -1.   1. ]
            [-1.  -0.5  1. ]
            [-0.5 -0.5  1. ]
            [ 0.  -0.5  1. ]
            [ 0.5 -0.5  1. ]
            [ 1.  -0.5  1. ]
            [-1.   0.   1. ]
            [-0.5  0.   1. ]
            [ 0.   0.   1. ]
            [ 0.5  0.   1. ]
            [ 1.   0.   1. ]
            [-1.   0.5  1. ]
            [-0.5  0.5  1. ]
            [ 0.   0.5  1. ]
            [ 0.5  0.5  1. ]
            [ 1.   0.5  1. ]
            [-1.   1.   1. ]
            [-0.5  1.   1. ]
            [ 0.   1.   1. ]
            [ 0.5  1.   1. ]
            [ 1.   1.   1. ]]
  Step3:
      Compute output by equation $$Output[i] = C_ * Theta[i]^T$$

Parameters

• theta (Variable) – A batch of affine transform parameters with shape [N, 2, 3].

• out_shape (Variable | list | tuple) – The shape of target output with format [N, C, H, W]. out_shape can be a Variable or a list or tuple.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The output with shape [N, H, W, 2].

Return type

Variable

Raises

ValueError – If the type of arguments is not supported.

Examples

import paddle.fluid as fluid
theta = fluid.layers.data(name="x", shape=[2, 3], dtype="float32")
out_shape = fluid.layers.data(name="y", shape=[-1], dtype="float32")
data = fluid.layers.affine_grid(theta, out_shape)

# or
data = fluid.layers.affine_grid(theta, [5, 3, 28, 28])

Create an auto-increase variable which will be automatically increased by 1 every mini-batch Return the run counter of the main program, default is started from 1.

Parameters

• counter_name (str) – The counter name, default is ‘@STEP_COUNTER@’.

• begin (int) – The first value of this counter.

• step (int) – The increment step between each execution.

Returns

The global run counter.

Return type

Variable

Examples

import paddle.fluid as fluid
global_step = fluid.layers.autoincreased_step_counter(
    counter_name='@LR_DECAY_COUNTER@', begin=0, step=1)

Batch Normalization Layer

Can be used as a normalizer function for conv2d and fully_connected operations. The required data format for this layer is one of the following:

1. NHWC [batch, in_height, in_width, in_channels]

2. NCHW [batch, in_channels, in_height, in_width]

Refer to Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift for more details.

\(input\) is the input features over a mini-batch.

When use_global_stats = True, the \(\mu_{\beta}\) and \(\sigma_{\beta}^{2}\) are not the statistics of one mini-batch. They are global (or running) statistics. (It usually got from the pre-trained model.) The training and testing (or inference) have the same behavior:

Parameters

• input (variable) – The rank of input variable can be 2, 3, 4, 5.

• act (string, Default None) – Activation type, linear|relu|prelu|…

• is_test (bool, Default False) – A flag indicating whether it is in test phrase or not.

• momentum (float, Default 0.9) – The value used for the moving_mean and moving_var computation. The updated formula is: \(moving\_mean = moving\_mean * momentum + new\_mean * (1. - momentum)\) \(moving\_var = moving\_var * momentum + new\_var * (1. - momentum)\) Default is 0.9.

• epsilon (float, Default 1e-05) – A value added to the denominator for numerical stability. Default is 1e-5.

• param_attr (ParamAttr|None) – The parameter attribute for Parameter scale of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|None) – The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• data_layout (string, default NCHW) – NCHW|NHWC

• in_place (bool, Default False) – Make the input and output of batch norm reuse memory.

• name (string, Default None) – A name for this layer(optional). If set None, the layer will be named automatically.

• moving_mean_name (string, Default None) – The name of moving_mean which store the global Mean. If it is set to None, batch_norm will save global mean with a random name, otherwise, batch_norm will save global mean with the string.

• moving_variance_name (string, Default None) – The name of the moving_variance which store the global Variance. If it is set to None, batch_norm will save global variance with a random name, otherwise, batch_norm will save global variance with the string.

• do_model_average_for_mean_and_var (bool, Default False) – Do model average for mean and variance or not.

• fuse_with_relu (bool) – if True, this OP performs relu after batch norm.

• use_global_stats (bool, Default False) – Whether to use global mean and variance. In inference or test mode, set use_global_stats to true or is_test to true, and the behavior is equivalent. In train mode, when setting use_global_stats True, the global mean and variance are also used during train period.

Returns

A tensor variable which is the result after applying batch normalization on the input.

Return type

Variable

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name='x', shape=[3, 7, 3, 7], dtype='float32', append_batch_size=False)
hidden1 = fluid.layers.fc(input=x, size=200, param_attr='fc1.w')
hidden2 = fluid.layers.batch_norm(input=hidden1)

Beam search is a classical algorithm for selecting candidate words in a machine translation task.

Refer to Beam search for more details.

This layer does the search in beams for one time step. Specifically, it selects the top-K candidate word ids of current step from ids according to their scores for all source sentences, where K is beam_size and ids, scores are predicted results from the computation cell. If ids is not set, it will be calculated out according to scores. Additionally, pre_ids and pre_scores are the output of beam_search at previous step, they are needed for special use to handle ended candidate translations.

Note that if is_accumulated is True, the scores passed in should be accumulated scores. Else, the scores are considered as the straightforward scores and will be transformed to the log field and accumulated the pre_scores in this operator. Length penalty should be done with extra operators before calculating the accumulated scores if needed.

Please see the following demo for a fully beam search usage example:

fluid/tests/book/test_machine_translation.py

Parameters

• pre_ids (Variable) – The LodTensor variable which is the output of beam_search at previous step. It should be a LodTensor with shape \((batch_size, 1)\) and lod \([[0, 1, ... , batch_size], [0, 1, ..., batch_size]]\) at the first step.

• pre_scores (Variable) – The LodTensor variable which is the output of beam_search at previous step.

• ids (Variable) – The LodTensor variable containing the candidates ids. Its shape should be \((batch_size \times beam_size, K)\), where \(K\) supposed to be beam_size.

• scores (Variable) – The LodTensor variable containing the accumulated scores corresponding to ids and its shape is the same as the shape of ids.

• beam_size (int) – The beam width used in beam search.

• end_id (int) – The id of end token.

• level (int, default 0) – It can be ignored and mustn’t change currently. It means the source level of lod, which is explained as following. The lod level of ids should be 2. The first level is source level which describes how many prefixes (branchs) for each source sentece (beam), and the second level is sentence level which describes how these candidates belong to the prefix. The paths linking prefixes and selected candidates are organized and reserved in lod.

• is_accumulated (bool, default True) – Whether the input score is accumulated scores.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

• return_parent_idx (bool) – Whether to return an extra Tensor variable preserving the selected_ids’ parent indice in pre_ids in output, which can be used to gather cell states at the next time step.

Returns

The LodTensor tuple containing the selected ids and the corresponding scores. If return_parent_idx is True, an extra Tensor variable preserving the selected_ids’ parent indice is included.

Return type

Variable

Examples

import paddle.fluid as fluid

# Suppose `probs` contains predicted results from the computation
# cell and `pre_ids` and `pre_scores` is the output of beam_search
# at previous step.
beam_size = 4
end_id = 1
pre_ids = fluid.layers.data(
    name='pre_id', shape=[1], lod_level=2, dtype='int64')
pre_scores = fluid.layers.data(
    name='pre_scores', shape=[1], lod_level=2, dtype='float32')
probs = fluid.layers.data(
    name='probs', shape=[10000], dtype='float32')
topk_scores, topk_indices = fluid.layers.topk(probs, k=beam_size)
accu_scores = fluid.layers.elementwise_add(
    x=fluid.layers.log(x=topk_scores),
    y=fluid.layers.reshape(pre_scores, shape=[-1]),
    axis=0)
selected_ids, selected_scores = fluid.layers.beam_search(
    pre_ids=pre_ids,
    pre_scores=pre_scores,
    ids=topk_indices,
    scores=accu_scores,
    beam_size=beam_size,
    end_id=end_id)

Beam Search Decode Layer. This layer constructs the full hypotheses for each source sentence by walking back along the LoDTensorArray ids whose lods can be used to restore the path in the beam search tree. Please see the following demo for a fully beam search usage example:

fluid/tests/book/test_machine_translation.py

Parameters

• ids (Variable) – The LodTensorArray variable containing the selected ids of all steps.

• scores (Variable) – The LodTensorArray variable containing the selected scores of all steps.

• beam_size (int) – The beam width used in beam search.

• end_id (int) – The id of end token.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The LodTensor pair containing the generated id sequences and the corresponding scores. The shapes and lods of the two LodTensor are same. The lod level is 2 and the two levels separately indicate how many hypotheses each source sentence has and how many ids each hypothesis has.

Return type

Variable

Examples

import paddle.fluid as fluid

# Suppose `ids` and `scores` are LodTensorArray variables reserving
# the selected ids and scores of all steps
ids = fluid.layers.create_array(dtype='int64')
scores = fluid.layers.create_array(dtype='float32')
finished_ids, finished_scores = fluid.layers.beam_search_decode(
    ids, scores, beam_size=5, end_id=0)

Add Bilinear Tensor Product Layer

This layer performs bilinear tensor product on two inputs. For example:

In this formula:

• \(x\): the first input contains M elements, shape is [batch_size, M].

• \(y\): the second input contains N elements, shape is [batch_size, N].

• \(W_{i}\): the i-th learned weight, shape is [M, N]

• \(out_{i}\): the i-th element of out, shape is [batch_size, size].

• \(y^\mathrm{T}\): the transpose of \(y_{2}\).

Parameters

• x (Variable) – 2-D input tensor with shape [batch_size, M]

• y (Variable) – 2-D input tensor with shape [batch_size, N]

• size (int) – The dimension of this layer.

• act (str, default None) – Activation to be applied to the output of this layer.

• name (str, default None) – The name of this layer.

• param_attr (ParamAttr, default None) – The parameter attribute for the learnable w. parameters/weights of this layer.

• bias_attr (ParamAttr, default None) – The parameter attribute for the bias of this layer. If it is set to False, no bias will be added to the output units. If it is set to None, the bias is initialized zero. Default: None.

Returns

A 2-D Tensor of shape [batch_size, size].

Return type

Variable

Examples

import paddle.fluid as fluid
layer1 = fluid.layers.data("t1", shape=[-1, 5], dtype="float32")
layer2 = fluid.layers.data("t2", shape=[-1, 4], dtype="float32")
tensor = fluid.layers.bilinear_tensor_product(x=layer1, y=layer2, size=1000)

Bayesian Personalized Ranking Loss Operator

This operator belongs to pairwise ranking loss. Label is the desired item. The loss at a given point in one session is defined as:

Learn more details by reading paper <session-based recommendations with recurrent neural networks>.

Parameters

• input (Variable|list) – a 2-D tensor with shape [N x D], where N is the batch size and D is the number of classes. This input is not probability but logits.

• label (Variable|list) – the ground truth which is a 2-D tensor. label is a tensor<int64> with shape [N x 1].

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically. Default: None.

Returns

A 2-D tensor with shape [N x 1], the bpr loss.

Examples

import paddle.fluid as fluid

neg_size = 10
label = fluid.layers.data(
          name="label", shape=[1], dtype="int64")
predict = fluid.layers.data(
          name="predict", shape=[neg_size + 1], dtype="float32")
cost = fluid.layers.bpr_loss(input=predict, label=label)

BRelu Activation Operator.

\(out = \max(\min(x, t_{min}), t_{max})\)

Parameters

• x (Variable) – Input of BRelu operator

• t_min (FLOAT|0.0) – The min marginal value of BRelu

• t_max (FLOAT|24.0) – The max marginal value of BRelu

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

Output of BRelu operator

Return type

output(Variable)

Examples:

import paddle.fluid as fluid
x = fluid.layers.data(name="x", shape=[2,3,16,16], dtype="float32")
y = fluid.layers.brelu(x, t_min=1.0, t_max=20.0)

Chunk Evaluator

This function computes and outputs the precision, recall and F1-score of chunk detection.

For some basics of chunking, please refer to Chunking with Support Vector Machines .

ChunkEvalOp computes the precision, recall, and F1-score of chunk detection, and supports IOB, IOE, IOBES and IO (also known as plain) tagging schemes. Here is a NER example of labeling for these tagging schemes:

====== ====== ======  =====  ==  ============   =====  ===== =====  ==  =========
       Li     Ming    works  at  Agricultural   Bank   of    China  in  Beijing.
====== ====== ======  =====  ==  ============   =====  ===== =====  ==  =========
IO     I-PER  I-PER   O      O   I-ORG          I-ORG  I-ORG I-ORG  O   I-LOC
IOB    B-PER  I-PER   O      O   B-ORG          I-ORG  I-ORG I-ORG  O   B-LOC
IOE    I-PER  E-PER   O      O   I-ORG          I-ORG  I-ORG E-ORG  O   E-LOC
IOBES  B-PER  E-PER   O      O   I-ORG          I-ORG  I-ORG E-ORG  O   S-LOC
====== ====== ======  =====  ==  ============   =====  ===== =====  ==  =========

There are three chunk types(named entity types) including PER(person), ORG(organization) and LOC(LOCATION), and we can see that the labels have the form <tag type>-<chunk type>.

Since the calculations actually use label ids rather than labels, extra attention should be paid when mapping labels to ids to make CheckEvalOp work. The key point is that the listed equations are satisfied by ids.

tag_type = label % num_tag_type
chunk_type = label / num_tag_type

where num_tag_type is the num of tag types in the tagging scheme, num_chunk_type is the num of chunk types, and tag_type get its value from the following table.

Scheme Begin Inside End   Single
 plain   0     -      -     -
 IOB     0     1      -     -
 IOE     -     0      1     -
 IOBES   0     1      2     3

Still use NER as example, assuming the tagging scheme is IOB while chunk types are ORG, PER and LOC. To satisfy the above equations, the label map can be like this:

B-ORG  0
I-ORG  1
B-PER  2
I-PER  3
B-LOC  4
I-LOC  5
O      6

It’s not hard to verify the equations noting that the num of chunk types is 3 and the num of tag types in IOB scheme is 2. For example, the label id of I-LOC is 5, the tag type id of I-LOC is 1, and the chunk type id of I-LOC is 2, which consistent with the results from the equations.

Parameters

• input (Variable) – prediction output of the network.

• label (Variable) – label of the test data set.

• chunk_scheme (str) – The labeling scheme indicating how to encode the chunks. Must be IOB, IOE, IOBES or plain. See the descriptionfor details

• num_chunk_types (int) – The number of chunk type. See the description for details

• excluded_chunk_types (list) – A list including chunk type ids indicating chunk types that are not counted. See the description for details

Returns

tuple containing: precision, recall, f1_score, num_infer_chunks, num_label_chunks, num_correct_chunks

Return type

tuple

Examples

import paddle.fluid as fluid

dict_size = 10000
label_dict_len = 7
sequence = fluid.layers.data(
    name='id', shape=[1], lod_level=1, dtype='int64')
embedding = fluid.layers.embedding(
    input=sequence, size=[dict_size, 512])
hidden = fluid.layers.fc(input=embedding, size=512)
label = fluid.layers.data(
    name='label', shape=[1], lod_level=1, dtype='int32')
crf = fluid.layers.linear_chain_crf(
    input=hidden, label=label, param_attr=fluid.ParamAttr(name="crfw"))
crf_decode = fluid.layers.crf_decoding(
    input=hidden, param_attr=fluid.ParamAttr(name="crfw"))
fluid.layers.chunk_eval(
    input=crf_decode,
    label=label,
    chunk_scheme="IOB",
    num_chunk_types=(label_dict_len - 1) / 2)

Clip Operator.

The clip operator limits the value of given input within an interval. The interval is specified with arguments ‘min’ and ‘max’:

$$ Out = min(max(X, min), max) $$

Parameters

• x (Variable) – (Tensor)The input of clip op.The number of dimensions must be between [1, 9]

• min (FLOAT) – (float)Minimum value, under which element is replaced by min

• max (FLOAT) – (float)Maximum value, above which element is replaced by max

• name (basestring|None) – Name of the output.

Returns

(Tensor)The output of clip op with shape as input(X)

Return type

out(Variable)

Examples

import paddle.fluid as fluid
input = fluid.layers.data(
    name='data', shape=[1], dtype='float32')
reward = fluid.layers.clip(x=input, min=-1.0, max=1.0)

ClipByNorm Operator.

This operator limits the L2 norm of the input \(X\) within \(max\_norm\). If the L2 norm of \(X\) is less than or equal to \(max\_norm\), \(Out\) will be the same as \(X\). If the L2 norm of \(X\) is greater than \(max\_norm\), \(X\) will be linearly scaled to make the L2 norm of \(Out\) equal to \(max\_norm\), as shown in the following formula:

$$ Out = \frac{max\_norm * X}{norm(X)}, $$

where \(norm(X)\) represents the L2 norm of \(X\).

Examples: .. code-block:: python

data = fluid.layer.data( name=’data’, shape=[2, 4, 6], dtype=’float32’) reshaped = fluid.layers.clip_by_norm( x=data, max_norm=0.5)

Parameters

• x (Variable) – (Tensor) The input of clip_by_norm op.The number of dimensions must be between [1, 9]

• max_norm (FLOAT) – (float) The maximum norm value

• name (basestring|None) – Name of the output.

Returns

(Tensor) The output of clip_by_norm op with shape as input(X)

Return type

out(Variable)

Examples

import paddle.fluid as fluid
input = fluid.layers.data(
    name='data', shape=[1], dtype='float32')
reward = fluid.layers.clip_by_norm(x=input, max_norm=1.0)

continuous_value_model layers

continuous value model(cvm). Now, it only considers show and click value in CTR project. We assume that input is an embedding vector with cvm_feature, whose shape is [N * D] (D is 2 + embedding dim). If use_cvm is True, it will log(cvm_feature), and output shape is [N * D]. If use_cvm is False, it will remove cvm_feature from input, and output shape is [N * (D - 2)].

This layer accepts a tensor named input which is ID after embedded(lod level is 1), cvm is a show_click info.

Parameters

• input (Variable) – a 2-D LodTensor with shape [N x D], where N is the batch size, D is 2 + the embedding dim. lod level = 1.

• cvm (Variable) – a 2-D Tensor with shape [N x 2], where N is the batch size, 2 is show and click.

• use_cvm (bool) – use cvm or not. if use cvm, the output dim is the same as input if don’t use cvm, the output dim is input dim - 2(remove show and click) (cvm op is a customized op, which input is a sequence has embed_with_cvm default, so we need an op named cvm to decided whever use it or not.)

Returns

A 2-D LodTensor with shape [N x D], if use cvm, D is equal to input dim, if don’t use cvm, D is equal to input dim - 2.

Return type

Variable

Examples

import paddle.fluid as fluid
input = fluid.layers.data(name="input", shape=[-1, 1], lod_level=1, append_batch_size=False, dtype="int64")#, stop_gradient=False)
label = fluid.layers.data(name="label", shape=[-1, 1], append_batch_size=False, dtype="int64")
embed = fluid.layers.embedding(
                  input=input,
                  size=[100, 11],
                  dtype='float32')
ones = fluid.layers.fill_constant_batch_size_like(input=label, shape=[-1, 1], dtype="int64", value=1)
show_clk = fluid.layers.cast(fluid.layers.concat([ones, label], axis=1), dtype='float32')
show_clk.stop_gradient = True
input_with_cvm = fluid.layers.continuous_value_model(embed, show_clk, True)

The convolution2D layer calculates the output based on the input, filter and strides, paddings, dilations, groups parameters. Input and Output are in NCHW format, where N is batch size, C is the number of channels, H is the height of the feature, and W is the width of the feature. Filter is in MCHW format, where M is the number of output image channels, C is the number of input image channels, H is the height of the filter, and W is the width of the filter. If the groups is greater than 1, C will equal the number of input image channels divided by the groups. Please refer to UFLDL’s convolution for more detials. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result.

For each input \(X\), the equation is:

Where:

• \(X\): Input value, a tensor with NCHW format.

• \(W\): Filter value, a tensor with MCHW format.

• \(\ast\): Convolution operation.

• \(b\): Bias value, a 2-D tensor with shape [M, 1].

• \(\sigma\): Activation function.

• \(Out\): Output value, the shape of \(Out\) and \(X\) may be different.

Example

• Input:

  Input shape: \((N, C_{in}, H_{in}, W_{in})\)

  Filter shape: \((C_{out}, C_{in}, H_f, W_f)\)

• Output:

  Output shape: \((N, C_{out}, H_{out}, W_{out})\)

Where

Parameters

• input (Variable) – The input image with [N, C, H, W] format.

• num_filters (int) – The number of filter. It is as same as the output image channel.

• filter_size (int|tuple|None) – The filter size. If filter_size is a tuple, it must contain two integers, (filter_size_H, filter_size_W). Otherwise, the filter will be a square.

• stride (int|tuple) – The stride size. If stride is a tuple, it must contain two integers, (stride_H, stride_W). Otherwise, the stride_H = stride_W = stride. Default: stride = 1.

• padding (int|tuple) – The padding size. If padding is a tuple, it must contain two integers, (padding_H, padding_W). Otherwise, the padding_H = padding_W = padding. Default: padding = 0.

• dilation (int|tuple) – The dilation size. If dilation is a tuple, it must contain two integers, (dilation_H, dilation_W). Otherwise, the dilation_H = dilation_W = dilation. Default: dilation = 1.

• groups (int) – The groups number of the Conv2d Layer. According to grouped convolution in Alex Krizhevsky’s Deep CNN paper: when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: groups=1.

• param_attr (ParamAttr|None) – The parameter attribute for learnable parameters/weights of conv2d. If it is set to None or one attribute of ParamAttr, conv2d will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with \(Normal(0.0, std)\), and the \(std\) is \((\frac{2.0 }{filter\_elem\_num})^{0.5}\). Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of conv2d. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv2d will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_cudnn (bool) – Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True

• act (str) – Activation type, if it is set to None, activation is not appended. Default: None

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically. Default: None

Returns

The tensor variable storing the convolution and non-linearity activation result.

Return type

Variable

Raises

ValueError – If the shapes of input, filter_size, stride, padding and groups mismatch.

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 32, 32], dtype='float32')
conv2d = fluid.layers.conv2d(input=data, num_filters=2, filter_size=3, act="relu")

Convlution2D transpose layer

The convolution2D transpose layer calculates the output based on the input, filter, and dilations, strides, paddings. Input(Input) and output(Output) are in NCHW format. Where N is batch size, C is the number of channels, H is the height of the feature, and W is the width of the feature. Parameters(dilations, strides, paddings) are two elements. These two elements represent height and width, respectively. The details of convolution transpose layer, please refer to the following explanation and references therein. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result.

For each input \(X\), the equation is:

Where:

• \(X\): Input value, a tensor with NCHW format.

• \(W\): Filter value, a tensor with MCHW format.

• \(\ast\): Convolution operation.

• \(b\): Bias value, a 2-D tensor with shape [M, 1].

• \(\sigma\): Activation function.

• \(Out\): Output value, the shape of \(Out\) and \(X\) may be different.

Example

• Input:

  Input shape: \((N, C_{in}, H_{in}, W_{in})\)

  Filter shape: \((C_{in}, C_{out}, H_f, W_f)\)

• Output:

  Output shape: \((N, C_{out}, H_{out}, W_{out})\)

Where

Parameters

• input (Variable) – The input image with [N, C, H, W] format.

• num_filters (int) – The number of the filter. It is as same as the output image channel.

• output_size (int|tuple|None) – The output image size. If output size is a tuple, it must contain two integers, (image_H, image_W). None if use filter_size, padding, and stride to calculate output_size. if output_size and filter_size are specified at the same time, They should follow the formula above.

• filter_size (int|tuple|None) – The filter size. If filter_size is a tuple, it must contain two integers, (filter_size_H, filter_size_W). Otherwise, the filter will be a square. None if use output size to calculate filter_size.

• padding (int|tuple) – The padding size. If padding is a tuple, it must contain two integers, (padding_H, padding_W). Otherwise, the padding_H = padding_W = padding. Default: padding = 0.

• stride (int|tuple) – The stride size. If stride is a tuple, it must contain two integers, (stride_H, stride_W). Otherwise, the stride_H = stride_W = stride. Default: stride = 1.

• dilation (int|tuple) – The dilation size. If dilation is a tuple, it must contain two integers, (dilation_H, dilation_W). Otherwise, the dilation_H = dilation_W = dilation. Default: dilation = 1.

• groups (int) – The groups number of the Conv2d transpose layer. Inspired by grouped convolution in Alex Krizhevsky’s Deep CNN paper, in which when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: groups = 1.

• param_attr (ParamAttr|None) – The parameter attribute for learnable parameters/weights of conv2d_transpose. If it is set to None or one attribute of ParamAttr, conv2d_transpose will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of conv2d_transpose. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv2d_transpose will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_cudnn (bool) – Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True.

• act (str) – Activation type, if it is set to None, activation is not appended. Default: None.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically. Default: True.

Returns

The tensor variable storing the convolution transpose result.

Return type

Variable

Raises

ValueError – If the shapes of input, filter_size, stride, padding and groups mismatch.

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 32, 32], dtype='float32')
conv2d_transpose = fluid.layers.conv2d_transpose(input=data, num_filters=2, filter_size=3)

Convlution3D Layer

The convolution3D layer calculates the output based on the input, filter and strides, paddings, dilations, groups parameters. Input(Input) and Output(Output) are in NCDHW format. Where N is batch size C is the number of channels, D is the depth of the feature, H is the height of the feature, and W is the width of the feature. Convlution3D is similar with Convlution2D but adds one dimension(depth). If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result.

For each input \(X\), the equation is:

In the above equation:

• \(X\): Input value, a tensor with NCDHW format.

• \(W\): Filter value, a tensor with MCDHW format.

• \(\ast\): Convolution operation.

• \(b\): Bias value, a 2-D tensor with shape [M, 1].

• \(\sigma\): Activation function.

• \(Out\): Output value, the shape of \(Out\) and \(X\) may be different.

Example

• Input:

  Input shape: \((N, C_{in}, D_{in}, H_{in}, W_{in})\)

  Filter shape: \((C_{out}, C_{in}, D_f, H_f, W_f)\)

• Output: Output shape: \((N, C_{out}, D_{out}, H_{out}, W_{out})\)

Where

Parameters

• input (Variable) – The input image with [N, C, D, H, W] format.

• num_filters (int) – The number of filter. It is as same as the output image channel.

• filter_size (int|tuple|None) – The filter size. If filter_size is a tuple, it must contain three integers, (filter_size_D, filter_size_H, filter_size_W). Otherwise, the filter will be a square.

• stride (int|tuple) – The stride size. If stride is a tuple, it must contain three integers, (stride_D, stride_H, stride_W). Otherwise, the stride_D = stride_H = stride_W = stride. Default: stride = 1.

• padding (int|tuple) – The padding size. If padding is a tuple, it must contain three integers, (padding_D, padding_H, padding_W). Otherwise, the padding_D = padding_H = padding_W = padding. Default: padding = 0.

• dilation (int|tuple) – The dilation size. If dilation is a tuple, it must contain three integers, (dilation_D, dilation_H, dilation_W). Otherwise, the dilation_D = dilation_H = dilation_W = dilation. Default: dilation = 1.

• groups (int) – The groups number of the Conv3d Layer. According to grouped convolution in Alex Krizhevsky’s Deep CNN paper: when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: groups=1

• param_attr (ParamAttr|None) – The parameter attribute for learnable parameters/weights of conv3d. If it is set to None or one attribute of ParamAttr, conv3d will create ParamAttr as param_attr. If it is set to None, the parameter is initialized with \(Normal(0.0, std)\), and the \(std\) is \((\frac{2.0 }{filter\_elem\_num})^{0.5}\). Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of conv3d. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv3d will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_cudnn (bool) – Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True

• act (str) – Activation type, if it is set to None, activation is not appended. Default: None.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically. Default: None.

Returns

The tensor variable storing the convolution and non-linearity activation result.

Return type

Variable

Raises

ValueError – If the shapes of input, filter_size, stride, padding and groups mismatch.

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 12, 32, 32], dtype='float32')
conv3d = fluid.layers.conv3d(input=data, num_filters=2, filter_size=3, act="relu")

Convlution3D transpose layer

The convolution3D transpose layer calculates the output based on the input, filter, and dilations, strides, paddings. Input(Input) and output(Output) are in NCDHW format. Where N is batch size, C is the number of channels, D is the depth of the feature, H is the height of the feature, and W is the width of the feature. Parameters(dilations, strides, paddings) are two elements. These two elements represent height and width, respectively. The details of convolution transpose layer, please refer to the following explanation and references therein. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result.

For each input \(X\), the equation is:

In the above equation:

• \(X\): Input value, a tensor with NCDHW format.

• \(W\): Filter value, a tensor with MCDHW format.

• \(\ast\): Convolution operation.

• \(b\): Bias value, a 2-D tensor with shape [M, 1].

• \(\sigma\): Activation function.

• \(Out\): Output value, the shape of \(Out\) and \(X\) may be different.

Example

• Input:

  Input shape: \((N, C_{in}, D_{in}, H_{in}, W_{in})\)

  Filter shape: \((C_{in}, C_{out}, D_f, H_f, W_f)\)

• Output:

  Output shape: \((N, C_{out}, D_{out}, H_{out}, W_{out})\)

Where

Parameters

• input (Variable) – The input image with [N, C, D, H, W] format.

• num_filters (int) – The number of the filter. It is as same as the output image channel.

• output_size (int|tuple|None) – The output image size. If output size is a tuple, it must contain three integers, (image_D, image_H, image_W). This parameter only works when filter_size is None.

• filter_size (int|tuple|None) – The filter size. If filter_size is a tuple, it must contain three integers, (filter_size_D, filter_size_H, filter_size_W). Otherwise, the filter will be a square. None if use output size to calculate filter_size.

• padding (int|tuple) – The padding size. If padding is a tuple, it must contain three integers, (padding_D, padding_H, padding_W). Otherwise, the padding_D = padding_H = padding_W = padding. Default: padding = 0.

• stride (int|tuple) – The stride size. If stride is a tuple, it must contain three integers, (stride_D, stride_H, stride_W). Otherwise, the stride_D = stride_H = stride_W = stride. Default: stride = 1.

• dilation (int|tuple) – The dilation size. If dilation is a tuple, it must contain three integers, (dilation_D, dilation_H, dilation_W). Otherwise, the dilation_D = dilation_H = dilation_W = dilation. Default: dilation = 1.

• groups (int) – The groups number of the Conv3d transpose layer. Inspired by grouped convolution in Alex Krizhevsky’s Deep CNN paper, in which when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: groups=1

• param_attr (ParamAttr|None) – The parameter attribute for learnable parameters/weights of conv3d_transpose. If it is set to None or one attribute of ParamAttr, conv3d_transpose will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of conv3d_transpose. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv3d_transpose will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_cudnn (bool) – Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True

• act (str) – Activation type, if it is set to None, activation is not appended. Default: None.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The tensor variable storing the convolution transpose result.

Return type

Variable

Raises

ValueError – If the shapes of input, filter_size, stride, padding and groups mismatch.

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 12, 32, 32], dtype='float32')
conv3d_transpose = fluid.layers.conv3d_transpose(input=data, num_filters=2, filter_size=3)

Cosine Similarity Operator

\(Out = \frac{X^T * Y}{(\sqrt{X^T * X} * \sqrt{Y^T * Y})}\)

The input X and Y must have the same shape, except that the 1st dimension of input Y could be just 1 (different from input X), which will be broadcasted to match the shape of input X before computing their cosine similarity.

Both the input X and Y can carry the LoD (Level of Details) information, or not. But the output only shares the LoD information with input X.

Parameters

• X (Variable) – The 1st input of cos_sim op.

• Y (Variable) – The 2nd input of cos_sim op.

Returns

the output of cosine(X, Y).

Return type

Variable

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name='x', shape=[3, 7], dtype='float32', append_batch_size=False)
y = fluid.layers.data(name='y', shape=[1, 7], dtype='float32', append_batch_size=False)
out = fluid.layers.cos_sim(x, y)

The crf_decoding operator reads the emission feature weights and the transition feature weights learned by the linear_chain_crf operator. It implements the Viterbi algorithm which is a dynamic programming algorithm for finding the most likely sequence of hidden states, called the Viterbi path, that results in a sequence of observed tags.

The output of this operator changes according to whether Input(Label) is given:

1. Input(Label) is given: This happens in training. This operator is used to co-work with the chunk_eval operator. When Input(Label) is given, the crf_decoding operator returns a row vector with shape [N x 1] whose values are fixed to be 0, indicating an incorrect prediction, or 1 indicating a tag is correctly predicted. Such an output is the input to chunk_eval operator.

2. Input(Label) is not given: This is the standard decoding process.

The crf_decoding operator returns a row vector with shape [N x 1] whose values range from 0 to maximum tag number - 1, Each element indicates an index of a predicted tag.

Parameters

• input (Variable) – (LoDTensor, default: LoDTensor<float>). A LoDTensor with shape [N x D] where N is the size of the mini-batch and D is the total tag number. This input is the unscaled emission weight matrix of the linear_chain_crf operator

• param_attr (ParamAttr) – The parameter attribute for training.

• label (Variable) – (LoDTensor, LoDTensor<int64_t>). The ground truth with shape [N x 1]. This input is optional. See more details in the operator’s comments

Returns

(LoDTensor, LoDTensor<int64_t>). The decoding results. What to return changes depending on whether the Input(Label) (the ground truth) is given. See more details in the operator’s comment

Return type

Variable

Examples

import paddle.fluid as fluid
images = fluid.layers.data(name='pixel', shape=[784], dtype='float32')
label = fluid.layers.data(name='label', shape=[1], dtype='int32')
hidden = fluid.layers.fc(input=images, size=2)
crf = fluid.layers.linear_chain_crf(input=hidden, label=label,
          param_attr=fluid.ParamAttr(name="crfw"))
crf_decode = fluid.layers.crf_decoding(input=hidden,
          param_attr=fluid.ParamAttr(name="crfw"))

Crop input into output, as specified by offsets and shape.

* Case 1:
    Given
        X = [[0, 1, 2, 0, 0]
             [0, 3, 4, 0, 0]
             [0, 0, 0, 0, 0]],
    and
        shape = [2, 2],
        offsets = [0, 1],
    output is:
        Out = [[1, 2],
               [3, 4]].
* Case 2:
    Given
        X = [[0, 1, 2, 5, 0]
             [0, 3, 4, 6, 0]
             [0, 0, 0, 0, 0]],
    and shape is tensor
        shape = [[0, 0, 0]
                 [0, 0, 0]]
    and
        offsets = [0, 1],

    output is:
        Out = [[1, 2, 5],
               [3, 4, 6]].

Parameters

• x (Variable) – The input tensor variable.

• shape (Variable|list/tuple of integer) – The output shape is specified by shape, which can a Variable or a list/tupe of integer. If a tensor Variable, it’s rank must be the same as x. This way is suitable for the case that the output shape may be changed each iteration. If a list/tupe of integer, it’s length must be the same as the rank of x

• offsets (Variable|list/tuple of integer|None) – Specifies the cropping offsets at each dimension. It can be a Variable or or a list/tupe of integers. If a tensor Variable, it’s rank must be the same as x. This way is suitable for the case that the offsets may be changed each iteration. If a list/tupe of integer, it’s length must be the same as the rank of x. If None, the offsets are 0 at each dimension.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The cropped tensor variable.

Return type

Variable

Raises

ValueError – If shape is not a list, tuple or Variable.

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name="x", shape=[3, 5], dtype="float32")
y = fluid.layers.data(name="y", shape=[2, 3], dtype="float32")
crop = fluid.layers.crop(x, shape=y)

# or
z = fluid.layers.data(name="z", shape=[3, 5], dtype="float32")
crop = fluid.layers.crop(z, shape=[-1, 2, 3])

Cross Entropy Layer

This layer computes the cross entropy between input and label. It supports both standard cross-entropy and soft-label cross-entropy loss computation.

1. One-hot cross-entropy:

   soft_label = False, Label[i, 0] indicates the class index for sample i:

   \[Y[i] = -\log(X[i, Label[i]])\]

2. Soft-label cross-entropy:

   soft_label = True, Label[i, j] indicates the soft label of class j for sample i:

   \[Y[i] = \sum_j{-Label[i, j] * log(X[i, j])}\]

   Please make sure that in this case the summation of each row of label equals one.

3. One-hot cross-entropy with vecterized label:

   As a special case of 2), when each row of ‘label’ has only one non-zero element which is equal to 1, soft-label cross-entropy degenerates to a one-hot cross-entropy with one-hot label representation.

Parameters

• input (Variable|list) – a 2-D tensor with shape [N x D], where N is the batch size and D is the number of classes. This input is a probability computed by the previous operator, which is almost always the result of a softmax operator.

• label (Variable|list) – the ground truth which is a 2-D tensor. When soft_label is set to False, label is a tensor<int64> with shape [N x 1]. When soft_label is set to True, label is a tensor<float/double> with shape [N x D].

• soft_label (bool) – a flag indicating whether to interpretate the given labels as soft labels. Default: False.

• ignore_index (int) – Specifies a target value that is ignored and does not contribute to the input gradient. Only valid if soft_label is set to False. Default: kIgnoreIndex

Returns

A 2-D tensor with shape [N x 1], the cross entropy loss.

Raises

ValueError – 1. the 1st dimension of input and label are not equal.

2. when soft_label == True, and the 2nd dimension of input and label are not equal.

3. when soft_label == False, and the 2nd dimension of label is not 1.

Examples

import paddle.fluid as fluid
classdim = 7
x = fluid.layers.data(name='x', shape=[3, 7], dtype='float32', append_batch_size=False)
label = fluid.layers.data(name='label', shape=[3, 1], dtype='float32', append_batch_size=False)
predict = fluid.layers.fc(input=x, size=classdim, act='softmax')
cost = fluid.layers.cross_entropy(input=predict, label=label)

This op is used to decode sequences by greedy policy by below steps:

1. Get the indexes of max value for each row in input. a.k.a. numpy.argmax(input, axis=0).

2. For each sequence in result of step1, merge repeated tokens between two blanks and delete all blanks.

A simple example as below:

Given:

input.data = [[0.6, 0.1, 0.3, 0.1],
              [0.3, 0.2, 0.4, 0.1],
              [0.1, 0.5, 0.1, 0.3],
              [0.5, 0.1, 0.3, 0.1],

              [0.5, 0.1, 0.3, 0.1],
              [0.2, 0.2, 0.2, 0.4],
              [0.2, 0.2, 0.1, 0.5],
              [0.5, 0.1, 0.3, 0.1]]

input.lod = [[4, 4]]

Computation:

step1: Apply argmax to first input sequence which is input.data[0:4]. Then we get:
       [[0], [2], [1], [0]]
step2: merge repeated tokens and remove blank which is 0. Then we get first output sequence:
       [[2], [1]]

Finally:

output.data = [[2],
               [1],
               [3]]

output.lod = [[2, 1]]

Parameters

• input (Variable) – (LoDTensor<float>), the probabilities of variable-length sequences, which is a 2-D Tensor with LoD information. It’s shape is [Lp, num_classes + 1], where Lp is the sum of all input sequences’ length and num_classes is the true number of classes. (not including the blank label).

• blank (int) – the blank label index of Connectionist Temporal Classification (CTC) loss, which is in thehalf-opened interval [0, num_classes + 1).

• name (str) – The name of this layer. It is optional.

Returns

CTC greedy decode result which is a 2-D tensor with shape [Lp, 1]. ‘Lp’ is the sum if all output sequences’ length. If all the sequences in result were empty, the result LoDTensor will be [-1] with LoD [[]] and dims [1, 1].

Return type

Variable

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name='x', shape=[8], dtype='float32')
cost = fluid.layers.ctc_greedy_decoder(input=x, blank=0)

Data Normalization Layer

Can be used as a normalizer function for conv2d and fully_connected operations. The required data format for this layer is one of the following:

1. NHWC [batch, in_height, in_width, in_channels]

2. NCHW [batch, in_channels, in_height, in_width]

\(input\) is the input features over a mini-batch.

Parameters

• input (variable) – The input variable which is a LoDTensor.

• act (string, Default None) – Activation type, linear|relu|prelu|…

• epsilon (float, Default 1e-05) –

• param_attr (ParamAttr) – The parameter attribute for Parameter scale.

• data_layout (string, default NCHW) – NCHW|NHWC

• in_place (bool, Default False) – Make the input and output of batch norm reuse memory.

• name (string, Default None) – A name for this layer(optional). If set None, the layer will be named automatically.

• moving_mean_name (string, Default None) – The name of moving_mean which store the global Mean.

• moving_variance_name (string, Default None) – The name of the moving_variance which store the global Variance.

• do_model_average_for_mean_and_var (bool, Default False) – Do model average for mean and variance or not.

Returns

A tensor variable which is the result after applying data normalization on the input.

Return type

Variable

Examples

import paddle.fluid as fluid

hidden1 = fluid.layers.data(name="hidden1", shape=[200])
hidden2 = fluid.layers.data_norm(name="hidden2", input=hidden1)

Deformable Convolution Layer

Compute 2-D deformable convolution on 4-D input. Given input image x, output feature map y, the deformable convolution operation can be expressed as follow:

Where \(\Delta p_k\) and \(\Delta m_k\) are the learnable offset and modulation scalar for the k-th location, respectively. Refer to Deformable ConvNets v2: More Deformable, Better Results .

Example

• Input:

  Input shape: \((N, C_{in}, H_{in}, W_{in})\)

  Filter shape: \((C_{out}, C_{in}, H_f, W_f)\)

  Offset shape: \((N, 2 * deformable\_groups * H_f * H_w, H_{in}, W_{in})\)

  Mask shape: \((N, deformable\_groups * H_f * H_w, H_{in}, W_{in})\)

• Output:

  Output shape: \((N, C_{out}, H_{out}, W_{out})\)

Where

Parameters

• input (Variable) – The input image with [N, C, H, W] format.

• offset (Variable) – The input coord offset of deformable convolution layer.

• Mask (Variable) – The input mask of deformable covolution layer.

• num_filters (int) – The number of filter. It is as same as the output image channel.

• filter_size (int|tuple|None) – The filter size. If filter_size is a tuple, it must contain two integers, (filter_size_H, filter_size_W). Otherwise, the filter will be a square.

• stride (int|tuple) – The stride size. If stride is a tuple, it must contain two integers, (stride_H, stride_W). Otherwise, the stride_H = stride_W = stride. Default: stride = 1.

• padding (int|tuple) – The padding size. If padding is a tuple, it must contain two integers, (padding_H, padding_W). Otherwise, the padding_H = padding_W = padding. Default: padding = 0.

• dilation (int|tuple) – The dilation size. If dilation is a tuple, it must contain two integers, (dilation_H, dilation_W). Otherwise, the dilation_H = dilation_W = dilation. Default: dilation = 1.

• groups (int) – The groups number of the deformable conv layer. According to grouped convolution in Alex Krizhevsky’s Deep CNN paper: when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: groups=1.

• deformable_groups (int) – The number of deformable group partitions. Default: deformable_groups = 1.

• im2col_step (int) – Maximum number of images per im2col computation; The total batch size should be divisable by this value or smaller than this value; if you face out of memory problem, you can try to use a smaller value here. Default: im2col_step = 64.

• param_attr (ParamAttr|None) – The parameter attribute for learnable parameters/weights of deformable conv. If it is set to None or one attribute of ParamAttr, deformable conv will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with \(Normal(0.0, std)\), and the \(std\) is \((\frac{2.0 }{filter\_elem\_num})^{0.5}\). Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of deformable conv layer. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv2d will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically. Default: None

Returns

The tensor variable storing the deformable convolution result.

Return type

Variable

Raises

ValueError – If the shapes of input, filter_size, stride, padding and groups mismatch.

Examples

import paddle.fluid as fluid
data = fluid.layers.data(name='data', shape=[3, 32, 32], dtype='float32')
offset = fluid.layers.data(name='offset', shape=[18, 32, 32], dtype='float32')
mask = fluid.layers.data(name='mask', shape=[9, 32, 32], dtype='float32')
out = fluid.layers.deformable_conv(input=data, offset=offset, mask=mask,
                                   num_filters=2, filter_size=3, padding=1)

Deformable PSROI Pooling Layer

Parameters

• input (Variable) – The input of Deformable PSROIPooling.The shape of input tensor is [N,C,H,W]. Where N is batch size,C is number of input channels,H is height of the feature, and W is the width of the feature.

• rois (Variable) – ROIs (Regions of Interest) to pool over.It should be a 2-D LoDTensor of shape (num_rois, 4), the lod level is 1. Given as [[x1, y1, x2, y2], …], (x1, y1) is the top left coordinates, and (x2, y2) is the bottom right coordinates.

• trans (Variable) – Offset of features on ROIs while pooling.The format is NCHW, where N is number of ROIs, C is number of channels, which indicate the offset distance in the x and y directions, H is pooled height, and W is pooled width.

• no_trans (bool) – Whether to add offset to get new value or not while roi pooling, which value is True or False. Default: False.

• spatial_scale (float) – Ratio of input feature map height (or width) to raw image height (or width). Equals the reciprocal of total stride in convolutional layers, Default: 1.0.

• group_size (list|tuple) – The number of groups which input channels are divided.(eg.number of input channels is k1*k2*(C+1), which k1 and k2 are group width and height and C+1 is number of output chanels. eg.(4, 6), which 4 is height of group and 6 is width of group. Default: [1, 1].

• pooled_height (integer) – The pooled output height. Default: 1.

• pooled_width (integer) – The pooled output width. Default: 1.

• part_size (list|tuple) – The height and width of offset, eg.(4, 6), which height is 4 and width is 6, Default: if None, default value is [pooled_height, pooled_width].

• sample_per_part (integer) – The number of samples in each bin. Default: 1.

• trans_std (float) – Coefficient of offset. Default: 0.1.

• position_sensitive (bool) – Whether to choose deformable psroi pooling mode or not. Default: False.

• name (str) – Name of layer. Default: None.

Returns

The tensor variable storing the deformable psroi pooling result.

Return type

Variable

Examples

import paddle.fluid as fluid
input = fluid.layers.data(name="input",
                          shape=[2, 192, 64, 64],
                          dtype='float32',
                          append_batch_size=False)
rois = fluid.layers.data(name="rois",
                         shape=[4],
                         dtype='float32',
                         lod_level=1)
trans = fluid.layers.data(name="trans",
                          shape=[2, 384, 64, 64],
                          dtype='float32',
                          append_batch_size=False)
x = fluid.layers.nn.deformable_roi_pooling(input=input,
                                             rois=rois,
                                             trans=trans,
                                             no_trans=False,
                                             spatial_scale=1.0,
                                             group_size=(1, 1),
                                             pooled_height=8,
                                             pooled_width=8,
                                             part_size=(8, 8),
                                             sample_per_part=4,
                                             trans_std=0.1,
                                             position_sensitive=False)

Dice loss for comparing the similarity of two batch of data, usually is used for binary image segmentation i.e. labels are binary. The dice loss can be defined as below equation:

Parameters

• input (Variable) – The predictions with rank>=2. The first dimension is batch size, and the last dimension is class number.

• label (Variable) – The groud truth with the same rank with input. The first dimension is batch size, and the last dimension is 1.

• epsilon (float) – The epsilon will be added to the numerator and denominator. If both input and label are empty, it makes sure dice is 1. Default: 0.00001

Returns

The dice loss with shape [1].

Return type

dice_loss (Variable)

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name='data', shape = [3, 224, 224, 2], dtype='float32')
label = fluid.layers.data(name='label', shape=[3, 224, 224, 1], dtype='float32')
predictions = fluid.layers.softmax(x)
loss = fluid.layers.dice_loss(input=predictions, label=label)

Computes dropout.

Drop or keep each element of x independently. Dropout is a regularization technique for reducing overfitting by preventing neuron co-adaption during training. The dropout operator randomly sets (according to the given dropout probability) the outputs of some units to zero, while others are remain unchanged.

dropout op can be removed from the program to make the program more efficient.

Parameters

• x (Variable) – The input tensor variable.

• dropout_prob (float) – Probability of setting units to zero.

• is_test (bool) – A flag indicating whether it is in test phrase or not.

• seed (int) – A Python integer used to create random seeds. If this parameter is set to None, a random seed is used. NOTE: If an integer seed is given, always the same output units will be dropped. DO NOT use a fixed seed in training.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

• dropout_implementation (string) –

  [‘downgrade_in_infer’(default)|’upscale_in_train’]

  1. downgrade_in_infer(default), downgrade the outcome at inference

     • train: out = input * mask

     • inference: out = input * (1.0 - dropout_prob)

     (mask is a tensor same shape with input, value is 0 or 1 ratio of 0 is dropout_prob)

  2. upscale_in_train, upscale the outcome at training time

     • train: out = input * mask / ( 1.0 - dropout_prob )

     • inference: out = input

     (mask is a tensor same shape with input, value is 0 or 1 ratio of 0 is dropout_prob)

Returns

A tensor variable is the shape with x.

Return type

Variable

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name="data", shape=[32, 32], dtype="float32")
droped = fluid.layers.dropout(x, dropout_prob=0.5)

Gated Recurrent Unit (GRU) Layer

if origin_mode is False, then the equation of a gru step is from paper Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling .

The formula is as follows:

if origin_mode is True then the equation is from paper Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation <https://arxiv.org/pdf/1406.1078.pdf>`_

The \(\odot\) is the element-wise product of the vectors. \(act_g\) is the update gate and reset gate activation function and \(sigmoid\) is usually used for it. \(act_c\) is the activation function for candidate hidden state and \(tanh\) is usually used for it.

Note that these \(W_{ux}x_{t}, W_{rx}x_{t}, W_{cx}x_{t}\) operations on the input \(x_{t}\) are NOT included in this operator. Users can choose to use fully-connect layer before GRU layer.

Parameters

• input (Variable) – The input of dynamic_gru layer, which supports variable-time length input sequence. The underlying tensor in this Variable is a matrix with shape \((T \times 3D)\), where \(T\) is the total time steps in this mini-batch, \(D\) is the hidden size.

• size (int) – The dimension of the gru cell.

• param_attr (ParamAttr|None) –

  The parameter attribute for the learnable hidden-hidden weight matrix. Note:

  • The shape of the weight matrix is \((T \times 3D)\), where \(D\) is the hidden size.

  • All elements in the weight matrix can be divided into two parts. The first part are weights of the update gate and reset gate with shape \((D \times 2D)\), and the second part are weights for candidate hidden state with shape \((D \times D)\).

  If it is set to None or one attribute of ParamAttr, dynamic_gru will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|bool|None) – The parameter attribute for the bias of GRU.Note that the bias with \((1 \times 3D)\) concatenates the bias in the update gate, reset gate and candidate calculations. If it is set to False, no bias will be applied to the update gate, reset gate and candidate calculations. If it is set to None or one attribute of ParamAttr, dynamic_gru will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• is_reverse (bool) – Whether to compute reversed GRU, default False.

• gate_activation (str) – The activation for update gate and reset gate. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “sigmoid”.

• candidate_activation (str) – The activation for candidate hidden state. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “tanh”.

• h_0 (Variable) – This is initial hidden state. If not set, default is zero. This is a tensor with shape (N x D), where N is the number of total time steps of input mini-batch feature and D is the hidden size.

Returns

The hidden state of GRU. The shape is \((T \times D)\), and sequence length is the same with the input.

Return type

Variable

Examples

import paddle.fluid as fluid

dict_dim, emb_dim = 128, 64
data = fluid.layers.data(name='sequence', shape=[1],
                         dtype='int32', lod_level=1)
emb = fluid.layers.embedding(input=data, size=[dict_dim, emb_dim])
hidden_dim = 512
x = fluid.layers.fc(input=emb, size=hidden_dim * 3)
hidden = fluid.layers.dynamic_gru(input=x, size=hidden_dim)

Long-Short Term Memory (LSTM) Operator.

The default implementation is diagonal/peephole connection (https://arxiv.org/pdf/1402.1128.pdf), the formula is as follows:

$$ i_t = \sigma(W_{ix}x_{t} + W_{ih}h_{t-1} + W_{ic}c_{t-1} + b_i) $$

$$ f_t = \sigma(W_{fx}x_{t} + W_{fh}h_{t-1} + W_{fc}c_{t-1} + b_f) $$

$$ \tilde{c_t} = act_g(W_{cx}x_t + W_{ch}h_{t-1} + b_c) $$

$$ o_t = \sigma(W_{ox}x_{t} + W_{oh}h_{t-1} + W_{oc}c_t + b_o) $$

$$ c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c_t} $$

$$ h_t = o_t \odot act_h(c_t) $$

• W terms denote weight matrices (e.g. \(W_{xi}\) is the matrix of weights from the input gate to the input), \(W_{ic}, W_{fc}, W_{oc}\) are diagonal weight matrices for peephole connections. In our implementation, we use vectors to represent these diagonal weight matrices. - The b terms denote bias vectors (\(b_i\) is the input gate bias vector). - \(\sigma\) is the non-line activations, such as logistic sigmoid function. - \(i, f, o\) and \(c\) are the input gate, forget gate, output gate, and cell activation vectors, respectively, all of which have the same size as the cell output activation vector \(h\). - The \(\odot\) is the element-wise product of the vectors. - \(act_g\) and \(act_h\) are the cell input and cell output activation functions and tanh is usually used for them. - \(\tilde{c_t}\) is also called candidate hidden state, which is computed based on the current input and the previous hidden state.

Set use_peepholes False to disable peephole connection. The formula is omitted here, please refer to the paper http://www.bioinf.jku.at/publications/older/2604.pdf for details.

Note that these \(W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}\) operations on the input \(x_{t}\) are NOT included in this operator. Users can choose to use fully-connect operator before LSTM operator.

Parameters

• input (Variable) – (LoDTensor) the first input is a LodTensor, which support variable-time length input sequence. The underlying tensor in this LoDTensor is a matrix with shape (T X 4D), where T is the total time steps in this mini-batch, D is the hidden size

• size (int) – 4 * hidden size.

• h_0 (Variable) – The initial hidden state is an optional input, default is zero. This is a tensor with shape (N x D), where N is the batch size and D is the hidden size.

• c_0 (Variable) – The initial cell state is an optional input, default is zero. This is a tensor with shape (N x D), where N is the batch size. h_0 and c_0 can be NULL but only at the same time.

• param_attr (ParamAttr|None) –

  The parameter attribute for the learnable hidden-hidden weights.

  • Weights = {\(W_{ch}, W_{ih}, W_{fh}, W_{oh}\)}

  • The shape is (D x 4D), where D is the hidden size.

  If it is set to None or one attribute of ParamAttr, dynamic_lstm will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|None) –

  The bias attribute for the learnable bias weights, which contains two parts, input-hidden bias weights and peephole connections weights if setting use_peepholes to True.

  1. use_peepholes = False - Biases = {\(b_c, b_i, b_f, b_o\)}. - The shape is (1 x 4D).

  2. use_peepholes = True - Biases = { \(b_c, b_i, b_f, b_o, W_{ic}, W_{fc}, W_{oc}\)}. - The shape is (1 x 7D).

  If it is set to None or one attribute of ParamAttr, dynamic_lstm will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_peepholes (bool) – (bool, default: True) whether to enable diagonal/peephole connections

• is_reverse (bool) – (bool, default: False) whether to compute reversed LSTM

• gate_activation (str) – (string, default: sigmoid)The activation for input gate, forget gate and output gate, sigmoid by default

• cell_activation (str) – (string, default: tanh)The activation for cell output, tanh by default

• candidate_activation (str) – (string, default: tanh)The activation for candidate hidden state, tanh by default

• dtype (str) – Data type. Choices = [“float32”, “float64”], default “float32”.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

Returns

The hidden state, and cell state of LSTM. The shape of both is (T x D), and lod is the same with the input.

Return type

tuple

Examples

import paddle.fluid as fluid
emb_dim = 256
vocab_size = 10000
hidden_dim = 512

data = fluid.layers.data(name='x', shape=[1],
             dtype='int32', lod_level=1)
emb = fluid.layers.embedding(input=data, size=[vocab_size, emb_dim], is_sparse=True)

forward_proj = fluid.layers.fc(input=emb, size=hidden_dim * 4,
                               bias_attr=False)

forward, _ = fluid.layers.dynamic_lstm(
    input=forward_proj, size=hidden_dim * 4, use_peepholes=False)

Dynamic LSTMP Layer

LSTMP (LSTM with recurrent projection) layer has a separate projection layer after the LSTM layer, projecting the original hidden state to a lower-dimensional one, which is proposed to reduce the number of total parameters and furthermore computational complexity for the LSTM, espeacially for the case that the size of output units is relative large (https://research.google.com/pubs/archive/43905.pdf).

The formula is as follows:

In the above formula:

• \(W\): Denotes weight matrices (e.g. \(W_{xi}\) is the matrix of weights from the input gate to the input).

• \(W_{ic}\), \(W_{fc}\), \(W_{oc}\): Diagonal weight matrices for peephole connections. In our implementation, we use vectors to represent these diagonal weight matrices.

• \(b\): Denotes bias vectors (e.g. \(b_i\) is the input gate bias vector).

• \(\sigma\): The activation, such as logistic sigmoid function.

• \(i, f, o\) and \(c\): The input gate, forget gate, output gate, and cell activation vectors, respectively, all of which have the same size as the cell output activation vector \(h\).

• \(h\): The hidden state.

• \(r\): The recurrent projection of the hidden state.

• \(\tilde{c_t}\): The candidate hidden state, whose computation is based on the current input and previous hidden state.

• \(\odot\): The element-wise product of the vectors.

• \(act_g\) and \(act_h\): The cell input and cell output activation functions and tanh is usually used for them.

• \(\overline{act_h}\): The activation function for the projection output, usually using identity or same as \(act_h\).

Set use_peepholes to False to disable peephole connection. The formula is omitted here, please refer to the paper http://www.bioinf.jku.at/publications/older/2604.pdf for details.

Note that these \(W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}\) operations on the input \(x_{t}\) are NOT included in this operator. Users can choose to use fully-connected layer before LSTMP layer.

Parameters

• input (Variable) – The input of dynamic_lstmp layer, which supports variable-time length input sequence. The underlying tensor in this Variable is a matrix with shape (T X 4D), where T is the total time steps in this mini-batch, D is the hidden size.

• size (int) – 4 * hidden size.

• proj_size (int) – The size of projection output.

• param_attr (ParamAttr|None) –

  The parameter attribute for the learnable hidden-hidden weight and projection weight.

  • Hidden-hidden weight = {\(W_{ch}, W_{ih}, W_{fh}, W_{oh}\)}.

  • The shape of hidden-hidden weight is (P x 4D), where P is the projection size and D the hidden size.

  • Projection weight = {\(W_{rh}\)}.

  • The shape of projection weight is (D x P).

  If it is set to None or one attribute of ParamAttr, dynamic_lstm will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

• bias_attr (ParamAttr|None) –

  The bias attribute for the learnable bias weights, which contains two parts, input-hidden bias weights and peephole connections weights if setting use_peepholes to True.

  1. use_peepholes = False

  • Biases = {\(b_c, b_i, b_f, b_o\)}.

  • The shape is (1 x 4D).

  2. use_peepholes = True

  • Biases = { \(b_c, b_i, b_f, b_o, W_{ic}, W_{fc}, W_{oc}\)}.

  • The shape is (1 x 7D).

  If it is set to None or one attribute of ParamAttr, dynamic_lstm will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

• use_peepholes (bool) – Whether to enable diagonal/peephole connections, default True.

• is_reverse (bool) – Whether to compute reversed LSTM, default False.

• gate_activation (str) – The activation for input gate, forget gate and output gate. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “sigmoid”.

• cell_activation (str) – The activation for cell output. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “tanh”.

• candidate_activation (str) – The activation for candidate hidden state. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “tanh”.

• proj_activation (str) – The activation for projection output. Choices = [“sigmoid”, “tanh”, “relu”, “identity”], default “tanh”.

• dtype (str) – Data type. Choices = [“float32”, “float64”], default “float32”.

• name (str|None) – A name for this layer(optional). If set None, the layer will be named automatically.

• h_0 (Variable) – The initial hidden state is an optional input, default is zero. This is a tensor with shape (N x D), where N is the batch size and D is the projection size.

• c_0 (Variable) – The initial cell state is an optional input, default is zero. This is a tensor with shape (N x D), where N is the batch size. h_0 and c_0 can be NULL but only at the same time.

• cell_clip (float) – If provided the cell state is clipped by this value prior to the cell output activation.

• proj_clip (float) – If num_proj > 0 and proj_clip is provided, then the projected values are clipped elementwise to within [-proj_clip, proj_clip].

Returns

A tuple of two output variable: the projection of hidden state, and cell state of LSTMP. The shape of projection is (T x P), for the cell state which is (T x D), and both LoD is the same with the input.

Return type

tuple

Examples

import paddle.fluid as fluid
dict_dim, emb_dim = 128, 64
data = fluid.layers.data(name='sequence', shape=[1],
                         dtype='int32', lod_level=1)
emb = fluid.layers.embedding(input=data, size=[dict_dim, emb_dim])
hidden_dim, proj_dim = 512, 256
fc_out = fluid.layers.fc(input=emb, size=hidden_dim * 4,
                         act=None, bias_attr=None)
proj_out, _ = fluid.layers.dynamic_lstmp(input=fc_out,
                                         size=hidden_dim * 4,
                                         proj_size=proj_dim,
                                         use_peepholes=False,
                                         is_reverse=True,
                                         cell_activation="tanh",
                                         proj_activation="tanh")

Edit distance operator computes the edit distances between a batch of hypothesis strings and their references. Edit distance, also called Levenshtein distance, measures how dissimilar two strings are by counting the minimum number of operations to transform one string into anthor. Here the operations include insertion, deletion, and substitution.

For example, given hypothesis string A = “kitten” and reference B = “sitting”, the edit distance is 3 for A will be transformed into B at least after two substitutions and one insertion:

“kitten” -> “sitten” -> “sittin” -> “sitting”

The input is a LoDTensor consisting of all the hypothesis strings with the total number denoted by batch_size, and the separation is specified by the LoD information. And the batch_size reference strings are arranged in order in the same way in the input LoDTensor.

The output contains the batch_size results and each stands for the edit distance for a pair of strings respectively. If Attr(normalized) is true, the edit distance will be divided by the length of reference string.

Parameters

• input (Variable) – The indices for hypothesis strings.

• label (Variable) – The indices for reference strings.

• normalized (bool, default True) – Indicated whether to normalize the edit distance by the length of reference string.

• ignored_tokens (list<int>, default None) – Tokens that should be removed before calculating edit distance.

• name (str) – The name of this layer. It is optional.

Returns

sequence-to-sequence edit distance in shape [batch_size, 1].

Return type

Variable

Examples

import paddle.fluid as fluid
x = fluid.layers.data(name='x', shape=[1], dtype='int64')
y = fluid.layers.data(name='y', shape=[1], dtype='int64')
cost, _ = fluid.layers.edit_distance(input=x, label=y)

cpu = fluid.core.CPUPlace()
exe = fluid.Executor(cpu)
exe.run(fluid.default_startup_program())

import numpy
x_ = numpy.random.randint(5, size=(2, 1)).astype('int64')
y_ = numpy.random.randint(5, size=(2, 1)).astype('int64')

print(x_)
print(y_)

x = fluid.create_lod_tensor(x_, [[2]], cpu)
y = fluid.create_lod_tensor(y_, [[2]], cpu)

outs = exe.run(feed={'x':x, 'y':y}, fetch_list=[cost.name])

print(outs)

Elementwise Add Operator

The equation is:

$$Out = X + Y$$

• \(X\): a tensor of any dimension.

• \(Y\): a tensor whose dimensions must be less than or equal to the dimensions of \(X\).

There are two cases for this operator:

1. The shape of \(Y\) is the same with \(X\).

2. The shape of \(Y\) is a continuous subsequence of \(X\).

For case 2:

1. Broadcast \(Y\) to match the shape of \(X\), where \(axis\) is the start dimension index for broadcasting \(Y\) onto \(X\).

2. If \(axis\) is -1 (default), \(axis = rank(X) - rank(Y)\).

3. The trailing dimensions of size 1 for \(Y\) will be ignored for the consideration of subsequence, such as shape(Y) = (2, 1) => (2).

For example:

shape(X) = (2, 3, 4, 5), shape(Y) = (,)
shape(X) = (2, 3, 4, 5), shape(Y) = (5,)
shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0

The inputs \(X\) and \(Y\) can carry the different LoD information. But the output only shares the LoD information with the input \(X\).

Parameters

• x – (Tensor), The first input tensor of elementwise op.

• y – (Tensor), The second input tensor of elementwise op.

• axis (INT) – (int, default -1). The start dimension index for broadcasting Y onto X.

• x_data_format (STRING) – (string, default NCHW) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• y_data_format (STRING) – (string, default “”) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• act (basestring|None) – Activation applied to the output.

• name (basestring|None) – Name of the output.

Returns

The output of elementwise op.

Examples

import paddle.fluid as fluid
# example 1: shape(x) = (2, 3, 4, 5), shape(y) = (2, 3, 4, 5)
x0 = fluid.layers.data(name="x0", shape=[2, 3, 4, 5], dtype='float32')
y0 = fluid.layers.data(name="y0", shape=[2, 3, 4, 5], dtype='float32')
z0 = fluid.layers.elementwise_add(x0, y0)

# example 2: shape(X) = (2, 3, 4, 5), shape(Y) = (5)
x1 = fluid.layers.data(name="x1", shape=[2, 3, 4, 5], dtype='float32')
y1 = fluid.layers.data(name="y1", shape=[5], dtype='float32')
z1 = fluid.layers.elementwise_add(x1, y1)

# example 3: shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
x2 = fluid.layers.data(name="x2", shape=[2, 3, 4, 5], dtype='float32')
y2 = fluid.layers.data(name="y2", shape=[4, 5], dtype='float32')
z2 = fluid.layers.elementwise_add(x2, y2, axis=2)

# example 4: shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
x3 = fluid.layers.data(name="x3", shape=[2, 3, 4, 5], dtype='float32')
y3 = fluid.layers.data(name="y3", shape=[3, 4], dtype='float32')
z3 = fluid.layers.elementwise_add(x3, y3, axis=1)

# example 5: shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
x4 = fluid.layers.data(name="x4", shape=[2, 3, 4, 5], dtype='float32')
y4 = fluid.layers.data(name="y4", shape=[2], dtype='float32')
z4 = fluid.layers.elementwise_add(x4, y4, axis=0)

# example 6: shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0
x5 = fluid.layers.data(name="x5", shape=[2, 3, 4, 5], dtype='float32')
y5 = fluid.layers.data(name="y5", shape=[2], dtype='float32')
z5 = fluid.layers.elementwise_add(x5, y5, axis=0)

Elementwise Div Operator

The equation is:

$$Out = X / Y$$

• \(X\): a tensor of any dimension.

• \(Y\): a tensor whose dimensions must be less than or equal to the dimensions of \(X\).

There are two cases for this operator:

1. The shape of \(Y\) is the same with \(X\).

2. The shape of \(Y\) is a continuous subsequence of \(X\).

For case 2:

1. Broadcast \(Y\) to match the shape of \(X\), where \(axis\) is the start dimension index for broadcasting \(Y\) onto \(X\).

2. If \(axis\) is -1 (default), \(axis = rank(X) - rank(Y)\).

3. The trailing dimensions of size 1 for \(Y\) will be ignored for the consideration of subsequence, such as shape(Y) = (2, 1) => (2).

For example:

shape(X) = (2, 3, 4, 5), shape(Y) = (,)
shape(X) = (2, 3, 4, 5), shape(Y) = (5,)
shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0

The inputs \(X\) and \(Y\) can carry the different LoD information. But the output only shares the LoD information with the input \(X\).

Parameters

• x – (Tensor), The first input tensor of elementwise op.

• y – (Tensor), The second input tensor of elementwise op.

• axis (INT) – (int, default -1). The start dimension index for broadcasting Y onto X.

• x_data_format (STRING) – (string, default NCHW) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• y_data_format (STRING) – (string, default “”) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• act (basestring|None) – Activation applied to the output.

• name (basestring|None) – Name of the output.

Returns

The output of elementwise op.

Examples

import paddle.fluid as fluid
# example 1: shape(x) = (2, 3, 4, 5), shape(y) = (2, 3, 4, 5)
x0 = fluid.layers.data(name="x0", shape=[2, 3, 4, 5], dtype='float32')
y0 = fluid.layers.data(name="y0", shape=[2, 3, 4, 5], dtype='float32')
z0 = fluid.layers.elementwise_div(x0, y0)

# example 2: shape(X) = (2, 3, 4, 5), shape(Y) = (5)
x1 = fluid.layers.data(name="x1", shape=[2, 3, 4, 5], dtype='float32')
y1 = fluid.layers.data(name="y1", shape=[5], dtype='float32')
z1 = fluid.layers.elementwise_div(x1, y1)

# example 3: shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
x2 = fluid.layers.data(name="x2", shape=[2, 3, 4, 5], dtype='float32')
y2 = fluid.layers.data(name="y2", shape=[4, 5], dtype='float32')
z2 = fluid.layers.elementwise_div(x2, y2, axis=2)

# example 4: shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
x3 = fluid.layers.data(name="x3", shape=[2, 3, 4, 5], dtype='float32')
y3 = fluid.layers.data(name="y3", shape=[3, 4], dtype='float32')
z3 = fluid.layers.elementwise_div(x3, y3, axis=1)

# example 5: shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
x4 = fluid.layers.data(name="x4", shape=[2, 3, 4, 5], dtype='float32')
y4 = fluid.layers.data(name="y4", shape=[2], dtype='float32')
z4 = fluid.layers.elementwise_div(x4, y4, axis=0)

# example 6: shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0
x5 = fluid.layers.data(name="x5", shape=[2, 3, 4, 5], dtype='float32')
y5 = fluid.layers.data(name="y5", shape=[2], dtype='float32')
z5 = fluid.layers.elementwise_div(x5, y5, axis=0)

Elementwise FloorDiv Operator

The equation is:

$$Out = X // Y$$

• \(X\): a tensor of any dimension.

• \(Y\): a tensor whose dimensions must be less than or equal to the dimensions of \(X\).

There are two cases for this operator:

1. The shape of \(Y\) is the same with \(X\).

2. The shape of \(Y\) is a continuous subsequence of \(X\).

For case 2:

1. Broadcast \(Y\) to match the shape of \(X\), where \(axis\) is the start dimension index for broadcasting \(Y\) onto \(X\).

2. If \(axis\) is -1 (default), \(axis = rank(X) - rank(Y)\).

3. The trailing dimensions of size 1 for \(Y\) will be ignored for the consideration of subsequence, such as shape(Y) = (2, 1) => (2).

For example:

shape(X) = (2, 3, 4, 5), shape(Y) = (,)
shape(X) = (2, 3, 4, 5), shape(Y) = (5,)
shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0

The inputs \(X\) and \(Y\) can carry the different LoD information. But the output only shares the LoD information with the input \(X\).

Parameters

• x – (Tensor), The first input tensor of elementwise op.

• y – (Tensor), The second input tensor of elementwise op.

• axis (INT) – (int, default -1). The start dimension index for broadcasting Y onto X.

• x_data_format (STRING) – (string, default NCHW) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• y_data_format (STRING) – (string, default “”) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• act (basestring|None) – Activation applied to the output.

• name (basestring|None) – Name of the output.

Returns

The output of elementwise op.

Examples

import paddle.fluid as fluid
# example 1: shape(x) = (2, 3, 4, 5), shape(y) = (2, 3, 4, 5)
x0 = fluid.layers.data(name="x0", shape=[2, 3, 4, 5], dtype='float32')
y0 = fluid.layers.data(name="y0", shape=[2, 3, 4, 5], dtype='float32')
z0 = fluid.layers.elementwise_floordiv(x0, y0)

# example 2: shape(X) = (2, 3, 4, 5), shape(Y) = (5)
x1 = fluid.layers.data(name="x1", shape=[2, 3, 4, 5], dtype='float32')
y1 = fluid.layers.data(name="y1", shape=[5], dtype='float32')
z1 = fluid.layers.elementwise_floordiv(x1, y1)

# example 3: shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
x2 = fluid.layers.data(name="x2", shape=[2, 3, 4, 5], dtype='float32')
y2 = fluid.layers.data(name="y2", shape=[4, 5], dtype='float32')
z2 = fluid.layers.elementwise_floordiv(x2, y2, axis=2)

# example 4: shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
x3 = fluid.layers.data(name="x3", shape=[2, 3, 4, 5], dtype='float32')
y3 = fluid.layers.data(name="y3", shape=[3, 4], dtype='float32')
z3 = fluid.layers.elementwise_floordiv(x3, y3, axis=1)

# example 5: shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
x4 = fluid.layers.data(name="x4", shape=[2, 3, 4, 5], dtype='float32')
y4 = fluid.layers.data(name="y4", shape=[2], dtype='float32')
z4 = fluid.layers.elementwise_floordiv(x4, y4, axis=0)

# example 6: shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0
x5 = fluid.layers.data(name="x5", shape=[2, 3, 4, 5], dtype='float32')
y5 = fluid.layers.data(name="y5", shape=[2], dtype='float32')
z5 = fluid.layers.elementwise_floordiv(x5, y5, axis=0)

Elementwise Max Operator

The equation is:

$$Out = max(X, Y)$$

• \(X\): a tensor of any dimension.

• \(Y\): a tensor whose dimensions must be less than or equal to the dimensions of \(X\).

There are two cases for this operator:

1. The shape of \(Y\) is the same with \(X\).

2. The shape of \(Y\) is a continuous subsequence of \(X\).

For case 2:

1. Broadcast \(Y\) to match the shape of \(X\), where \(axis\) is the start dimension index for broadcasting \(Y\) onto \(X\).

2. If \(axis\) is -1 (default), \(axis = rank(X) - rank(Y)\).

3. The trailing dimensions of size 1 for \(Y\) will be ignored for the consideration of subsequence, such as shape(Y) = (2, 1) => (2).

For example:

shape(X) = (2, 3, 4, 5), shape(Y) = (,)
shape(X) = (2, 3, 4, 5), shape(Y) = (5,)
shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0

The inputs \(X\) and \(Y\) can carry the different LoD information. But the output only shares the LoD information with the input \(X\).

Parameters

• x – (Tensor), The first input tensor of elementwise op.

• y – (Tensor), The second input tensor of elementwise op.

• axis (INT) – (int, default -1). The start dimension index for broadcasting Y onto X.

• x_data_format (STRING) – (string, default NCHW) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• y_data_format (STRING) – (string, default “”) Only used in mkldnnAn optional string from: “NHWC”, “NCHW”, “NCHW16C”, “NCHW8C”. Defaults to “”. Specify the data format of the output data, the input will be transformed automatically.

• act (basestring|None) – Activation applied to the output.

• name (basestring|None) – Name of the output.

Returns

The output of elementwise op.

Examples

import paddle.fluid as fluid
# example 1: shape(x) = (2, 3, 4, 5), shape(y) = (2, 3, 4, 5)
x0 = fluid.layers.data(name="x0", shape=[2, 3, 4, 5], dtype='float32')
y0 = fluid.layers.data(name="y0", shape=[2, 3, 4, 5], dtype='float32')
z0 = fluid.layers.elementwise_max(x0, y0)

# example 2: shape(X) = (2, 3, 4, 5), shape(Y) = (5)
x1 = fluid.layers.data(name="x1", shape=[2, 3, 4, 5], dtype='float32')
y1 = fluid.layers.data(name="y1", shape=[5], dtype='float32')
z1 = fluid.layers.elementwise_max(x1, y1)

# example 3: shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5), with axis=-1(default) or axis=2
x2 = fluid.layers.data(name="x2", shape=[2, 3, 4, 5], dtype='float32')
y2 = fluid.layers.data(name="y2", shape=[4, 5], dtype='float32')
z2 = fluid.layers.elementwise_max(x2, y2, axis=2)

# example 4: shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1
x3 = fluid.layers.data(name="x3", shape=[2, 3, 4, 5], dtype='float32')
y3 = fluid.layers.data(name="y3", shape=[3, 4], dtype='float32')
z3 = fluid.layers.elementwise_max(x3, y3, axis=1)

# example 5: shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0
x4 = fluid.layers.data(name="x4", shape=[2, 3, 4, 5], dtype='float32')
y4 = fluid.layers.data(name="y4", shape=[2], dtype='float32')
z4 = fluid.layers.elementwise_max(x4, y4, axis=0)

# example 6: shape(X) = (2, 3, 4, 5), shape(Y) = (2, 1), with axis=0
x5 = fluid.layers.data(name="x5", shape=[2, 3, 4, 5], dtype='float32')
y5 = fluid.layers.data(name="y5", shape=[2], dtype='float32')
z5 = fluid.layers.elementwise_max(x5, y5, axis=0)