《神经网络编程——使用PyTorch进行深度学习》课程代码（一）

You can find this course at YouTube or DEEPLIZARD. While in China, those videos were added Chinese subtitles and transfered to Bilibili by an uploader, here is the link Deeplizard《Pytorch神经网络高效入门教程》中文字幕版.

I write this blog following the teacher in the video and you can share it without my permission.

import torch
import numpy as np

Check the installation

print(torch.__version__)

1.7.0

torch.cuda.is_available()

True

Tensor Class

t = torch.Tensor()
type(t)

torch.Tensor

Tensor attributes

print(t.dtype)
print(t.device)
print(t.layout)

torch.float32
cpu
torch.strided

device = torch.device('cuda:0')
device

device(type='cuda', index=0)

The Concepts of Tensor

dd = [
    [
        [1,2,1,2],
        [3,4,3,4]
    ],
    [
        [5,6,5,6],
        [7,8,7,8]
    ],
    [
        [9,10,9,10],
        [11,12,11,12]
    ]
]

t = torch.tensor(dd)

type(t)

torch.Tensor

The shape of a tensor = tensor’s size

t.shape

torch.Size([3, 2, 4])

The rank of a tensor = the length of tensor’s shape = the number of tensor’s axis

len(t.shape)

3

Reconstruct a tensor using original tensor, note that $3 \times 2 \times 4$ must = $1 \times 24$

t.reshape(1,24).shape

torch.Size([1, 24])

Tips

t1 = torch.tensor([1,2,3])
t2 = torch.tensor([1.,2.,3.])

t1.dtype

torch.int64

t2.dtype

torch.float32

t3 = t1 + t2
t3

tensor([2., 4., 6.])

t3.dtype

torch.float32

In older versions, pytorch don’t allow tensors with different data type to operate with each other, for example, t1(torch.int64) + t2(forch.float32) is not allowed. if you do, you’ll get an exception like:

Expected object of type torch.LongTensor but found type torch.FloatTensor forargument #3 'other'

Note that tensors from different devices can’t operate with each other, too.

t1 = torch.tensor([1,2,3])
t2 = t1.cuda()

t1.device

device(type='cpu')

t2.device

device(type='cuda', index=0)

t1 + t2

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-20-9ac58c83af08> in <module>
----> 1 t1 + t2


RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Creation options using data

1. torch.Tensor(data)
2. torch.tensor(data)
3. torch.as_tensor(data)
4. torch.from_numpy(data)

data1 = torch.Tensor([1,2,3])
print(data1)
data1.dtype

tensor([1., 2., 3.])

torch.float32

data2 = torch.tensor([1,2,3])
print(data2)
data2.dtype

tensor([1, 2, 3])

torch.int64

See the difference? When useing torch.Tensor(), it will call the constructer of the Tensor class while torch.tensor() is the factory function. The data type of them are different because the int data type data that you input is transformed into float data type by Tensor() but tensor() doesn’t do this transformation.

data = np.array([1,2,3])
type(data)

numpy.ndarray

torch.Tensor(data)

tensor([1., 2., 3.])

torch.tensor(data)

tensor([1, 2, 3], dtype=torch.int32)

torch.as_tensor(data)

tensor([1, 2, 3], dtype=torch.int32)

torch.from_numpy(data)

tensor([1, 2, 3], dtype=torch.int32)

torch.eye(3)

tensor([[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]])

torch.zeros(2,2)

tensor([[0., 0.],
        [0., 0.]])

torch.ones(2,2)

tensor([[1., 1.],
        [1., 1.]])

torch.rand(2,2)

tensor([[0.2750, 0.7079],
        [0.8626, 0.8727]])

Best Options

data = np.array([1, 2, 3])

t1 = torch.Tensor(data)
t2 = torch.tensor(data)
t3 = torch.as_tensor(data)
t4 = torch.from_numpy(data)

We have three factory funtions: tensor, as_tensor, from_numpy, and one contructor: Tensor.

print(t1)
print(t2)
print(t3)
print(t4)

tensor([1., 2., 3.])
tensor([1, 2, 3], dtype=torch.int32)
tensor([1, 2, 3], dtype=torch.int32)
tensor([1, 2, 3], dtype=torch.int32)

print(t1.dtype)
print(t2.dtype)
print(t3.dtype)
print(t4.dtype)

torch.float32
torch.int32
torch.int32
torch.int32

This difference occurs because the constructor uses the global default data type value when constructing a tensor while the factory functions infer the data type.

torch.get_default_dtype()

torch.float32

torch.tensor(np.array([1, 2, 3]), dtype=torch.float64)

tensor([1., 2., 3.], dtype=torch.float64)

Sharing vs Copying

data = np.array([1, 2, 3])
data

array([1, 2, 3])

t1 = torch.Tensor(data)
t2 = torch.tensor(data)
t3 = torch.as_tensor(data)
t4 = torch.from_numpy(data)

data[0] = 0
data[1] = 0
data[2] = 0

print(t1)
print(t2)
print(t3)
print(t4)

tensor([1., 2., 3.])
tensor([1, 2, 3], dtype=torch.int32)
tensor([0, 0, 0], dtype=torch.int32)
tensor([0, 0, 0], dtype=torch.int32)

Which methods should we choose?

The best option is the lowercase T function torch.tensor().
The second option is torch.as_tensor().

1. Since numpy.ndarray objects are allocated on the CPU, the as_tensor() function must copy the data from the CPU to the GPU when a GPU is being used.
2. The memory sharing of as_tensor() doesn’t work with built-in Python data structures like lists.
3. The as_tensor() call requires developer knowledge of the sharing feature. This is necessary so we don’t inadvertently make an unwanted change in the underlying data without realizing the change impacts multiple objects.
4. The as_tensor() performance improvement will be greater if there are a lot of back and forth operations between numpy.ndarray objects and tensor objects. However, if there is just a single load operation, there shouldn’t be much impact from a performance perspective.