⚕️Brief introduction of the Tensor in PyTorch

Tensor is a specialized data structure that is very similar to arrays and matrices. We can use it to encode the input and output of the model. Tensors can run on GPUs and other hardware.

Initializing a Tensors

Tensors can be initialized in various ways,

# Import the library
import torch

# Directly from data
data=[[1,2],
      [2,3]]
t_data = torch.tensor(data)

# From Numpy
import numpy as np
np_data = np.array(data)
t_np =  torch.from_numpy(np_data)

# From other tensors
# In this way, it will retains same properties
t_ones = torch.ones_like(t_data) 
# override the datatype
t_random = torch.rand_like(t_data,dtype=torch.float) 

# With static shape
shape = (2,3,)
rand_t = torch.rand(shape)
ones_t = torch.ones(shape)
zeros_t = torch.zeros(shape)

Careful: If you initialize the tensors from Numpy, they will share the same underlying memory, which means that if changing the numpy array, the tensor will change too.

Attributes of a Tensor

We can get some attributes of a tensor by code followed

t_data.shape # Get the shape of the tensor
t_data.dtype # Get the dtype
t_data.device #  return the device(CPU or GPU) for this tensor

Operation on Tensor

There are over 100 operations for tensors, including arithmetic, linear algebra, matrix manipulation, and sampling. Each of these operations can be run on GUP, but by default it’s in CPU, we can move by .to like the following:

if torch.cuda.is_available():
    t_data = t_data.to('cuda')

Here are some common operations:

1. Indexing and Slicing

   t_data[0] # get the first row of tensor
   t_data[:,0] # get the first column of tensor
   t_data[...,-1] # get the last column of tensor
   t_data[:,1]=0 # change secound column to zero
   

2. Joining tensors

   we can concatenate a sequence of tensors by a given dimension torch.cat([t_data,t_data],dim=1)

3. Arithmetic operations

   # computes the matrix multiplication
   tensor = t_data @ t_data.T
   tensor = t_data.matmul(t_data.T)
   torch.matmul(t_data,t_data.T,out = tensor)
   
   # computes the element-wise product
   tensor = t_data * t_data.T
   tensor = t_data.mul(t_data.T)
   torch.mul(t_data,t_data.T,out = tensor)
   

4. Single-element tensors

   If you have one-element tensor, can convert it to python value by t_data.item()

5. In place operations

   Sometime we would like to change tensors them self, instead of return an answer. In order to do this we can add _ behind the function.

   For example:t_data.add_(5)will add five for all element it self.