Simple Introduction#
Expand
Overall Plan#
Expand
The plan is based on existing tutorials on the internet, using virtualization to build a personal streaming media service for practice, aiming to achieve automated watching of episodes while learning some virtualization and server-related knowledge.
The overall framework of the streaming media service is as follows:
- proxmox supports virtualization operation
- docker or k8s serves as the basic platform for service operation
- qbittorrent as the downloader
- jackett acts as the subscription indexer
- radarr and sonarr play the role of movie and series scrapers
- overseerr provides scraping integration services
- plex and emby are responsible for media center services
- bazarr matches subtitle information
The above is based on the X99 platform's 2U server, but it doesn't necessarily have to be a rack-mounted server; other DIY devices can also implement this plan as long as there are enough cores and sufficient memory to handle virtualization.
The projects used can all be found in the world's largest open-source community.
Flowchart#
Q&A:#
Expand
Q: Why not use NAS products like unraid, black 群,openmediavault, turnas, etc.?
A: It must be admitted that NAS products have better ecological support, a complete operating experience, and relatively low learning costs, especially black 群 unraid. However, using NAS products reduces some playability 😑. Choosing virtualization, although more cumbersome, brings various problems and solutions that I find to be the greatest joy of tinkering, and it also allows for a lot of knowledge in server operation and maintenance 🤩.
Q: What are the pros and cons of this plan in terms of viewing experience?
A: The advantage is that it is relatively simple to configure, and the interface is beautiful and easy to operate. Once perfected, you only need to click request on the shows you want to watch in overseerr to push them to qb for download. No other operations are required throughout the process, but the drawbacks are also obvious. Sonarr does not support Chinese, leading to anime, domestic, and Japanese and Korean dramas being renamed in English, which makes long file names not very aesthetically pleasing when displayed on Emby. Newer resources can be effectively supported, but if there are no PT site resources, it is difficult to find older film and television resources. (This issue exists in any plan 😟)
Q: What about other expansions?
A: For example, downloaders like utorrent can be expanded, and media centers can add kodi, jillyfin, etc. Mobile subscriptions for videos can also be achieved through webhook message pushing. For anime series, AutoBangumi can be used for subscription methods, etc. In short, there is still a lot of room for play; this plan is just a foundation or a beginning, and there is much exploration space ahead 😉.
Q: Other automation plans?
A: The core experience of automated viewing lies in scraping video information, allowing the media center to recognize film information. The focus of scraping is on renaming video files or their parent directories. From these perspectives, tools like nastools and tMM can serve as alternatives. For subtitles, ChineseSubFinder can be a substitute. In terms of virtualization, proxmox can also use exsi as a replacement.
Virtualization Underlying System#
Some notes about proxmox
Expand
- Image file download address: proxmox VE
- USB boot tool: Rufus or Ventoy, the latter is more recommended
- Proxmox does not have a default operating page; you can control it via the web interface through the connection provided after system installation, defaulting to https://ip:8006/
- Change source
Enterprise source
## Remove the built-in source rm /etc/apt/sources.list.d/pve-enterprise.list ## Add the official non-subscription source (choose one between this and the domestic non-subscription source below) echo 'deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription' >> /etc/apt/sources.list.d/pve-no-subscription.list ## This is the domestic non-subscription source echo 'deb http://mirrors.ustc.edu.cn/proxmox/debian/pve bullseye pve-no-subscription' >> /etc/apt/sources.list.d/pve-no-subscription.list
Software source
## Edit the source file: /etc/apt/sources.list cp /etc/apt/sources.list /etc/apt/sources.list.bak && vim /etc/apt/sources.list ## Delete all content in the file or comment it out with #, and add the following content deb https://mirrors.aliyun.com/debian bullseye main contrib deb https://mirrors.aliyun.com/debian bullseye-updates main contrib deb https://mirrors.aliyun.com/debian-security bullseye-security main contrib
- Multiple disks can be configured in RAID 5 to provide sufficient storage space for virtual machines while ensuring relatively reliable data storage.
Virtual Machine#
About managing Linux and necessary configurations in the plan
Expand
At least 3 virtual machines need to be created:
157.20.20.1 --> jumpserver --> Bastion host for remote operation and maintenance
157.20.20.2 --> nfs --> NFS file sharing system
157.20.20.3 --> docker --> Dedicated container for deploying streaming media
If choosing the k8s method, at least 2 more virtual machines need to be virtualized:
157.20.20.3 --> k8s-master --> k8s master node
157.20.20.4 --> k8s-node01 --> k8s worker node
157.20.20.4 --> k8s-node02 --> k8s worker node
Among them, the jumpserver needs at least 4C8G resources, and other virtual machines should allocate at least 2 cores, with memory allocated as needed, without special requirements. If it is a k8s environment, the master node needs at least 2C6G, and the node nodes need at least 2C4G.
The reason for creating multiple virtual machines: to divide space according to function, avoiding all components becoming unavailable due to improper operations that damage server files. Once this happens, repair and maintenance will become exceptionally troublesome.
Creating Virtual Machines#
ISO images need to be uploaded to the local pve for installing the basic operating system.
Various basic image download addresses: CentOS7, Rocky Linux, Debian
Click "Create Virtual Machine" in the upper right corner and select the appropriate image for installation.
Installing Jumpserver on the Virtual Machine#
Simple Introduction
Jumpserver is an open-source bastion host product under Feizhi Cloud, which is much more convenient than various SSH and SCP tools. Official website
Deployment
curl -sSL https://resource.fit2cloud.com/jumpserver/jumpserver/releases/latest/download/quick_start.sh | bash
████████████████████████████████████████ 100%
[Success]: download install script to /opt/jumpserver-installer-v3.2.0
[Info]: Start executing the installation script.
[Info]: In an automated script deployment, note the message prompts on the screen.
████████████████████████████████████████ 100%
[Success]: The Installation is Complete.
For more commands, you can enter jmsctl --help to view help information.
After installation, access it in the browser at http://157.20.20.1 and log in with the default account admin/admin.
Creating the First Asset
Add assets in Asset Management --> Asset List.
Create authorization rules for user asset authorization in Permission Management --> Asset Authorization.
After authorization is complete, you can enter the console in the upper right corner to log in to the virtual machine and perform other operations, which is more flexible and convenient than SSH tools. At this point, the jumpserver deployment is complete.
Creating NFS Shared File Server#
Create a second Linux virtual machine and allocate sufficient disk space. Plan data space and media space in advance for easier management and to avoid confusion.
Create media and storage folders in the system, respectively serving as storage space and media space, and mount the disks properly.
## Confirm the space to be mounted
$ lsblk
## Create media and storage folders
$ cd /mnt && mkdir media && mkdir storage
## Format and mount the disk to the folder for the operating system to access
## File system format can be xfs or ext4, etc.
$ mkfs.xfs /dev/sdb
$ mkfs.xfs /dev/sdc
$ mount /dev/sdb /mnt/storage && mount /dev/sdc /mnt/storage
## Modify /etc/fstab to ensure automatic mounting at boot
$ vim /etc/fstab
/dev/sdb /mnt/storage xfs defaults 0 2
/dev/sdc /mnt/media xfs defaults 0 2
Deploy the NFS server, taking Rocky Linux as an example.
- Install NFS server components:
$ yum install nfs-utils
- Set permissions for the directory to be shared:
$ chmod 777 /mnt/storage
$ chmod 777 /mnt/media
- Configure NFS access permission file /etc/exports. For example:
$ vim /etc/exports
/mnt/storage 157.20.20.0/24(rw,sync,no_subtree_check)
/mnt/media 157.20.20.0/24(rw,sync,no_subtree_check)
## This will provide read and write access to clients in the 157.20.20.0/24 subnet
- Start the NFS service:
$ systemctl start nfs-server
$ systemctl enable nfs-server # Set to start automatically at boot
- Open the NFS connection port in the firewall:
$ firewall-cmd --permanent --add-service=nfs
$ firewall-cmd --permanent --add-port=2049/tcp
$ firewall-cmd --reload
- Test the NFS configuration:
$ showmount -e localhost
/mnt/storage 157.20.20.0/24
/mnt/media 157.20.20.0/24
Tip
If you modify the contents of exports, you need to re-export it as follows:$ exportfs -r $ systemctl restart nfs-server
Docker Server#
Create a third Linux virtual machine and deploy the NFS server, taking Rocky Linux as an example.
$ yum install -y yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
$ yum makecache fast && yum -y install docker-ce
$ systemctl start docker && systemctl enable docker
Upload the docker-compose file downloaded from GitHub through the bastion host's file management. Download link
The uploaded file is located in /tmp
by default.
$ cp /tmp/docker-compose-linux-x86_64 /usr/local/sbin/docker-compose
$ chmod +x /usr/local/sbin/docker-compose
After completion, you can check the versions of docker and docker-compose.
$ docker version
$ docker-compose version
Quick Deployment of K8s Environment#
Open-source project: KuboardSpray-github
This project can be directly deployed on the jumpserver virtual machine. Note that port 80 is already occupied, so you need to change the port when mapping.
docker run -d \
--restart=unless-stopped \
--name=kuboard-spray \
-p 8081:80/tcp \
-e TZ=Asia/Shanghai \
-v /var/run/docker.sock:/var/run/docker.sock \
-v ~/kuboard-spray-data:/data \
eipwork/kuboard-spray:latest-amd64
# If you can't grab this image, you can try this backup address:
# swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard-spray:latest-amd64
Access it via http://157.20.20.1:8081, with the default account admin/Kuboard123.
Using kuboard-spray, you can manage the virtual machines from 157.20.20.3 to 157.20.20.6 (specifically see which virtual machines to use as k8s environment nodes).
It is worth noting that a single server can only simulate a k8s cluster for testing and learning purposes. A real k8s cluster should have multiple master nodes and worker nodes distributed across different physical servers to ensure that a single machine failure does not affect cluster operation.
For detailed installation and deployment reference, see the kuboard-spray official documentation.
For optimization reference:
To be updated
Downloader#
About deploying and using qbittorrent via docker or k8s
Expand
Deployment#
Docker Method#
Mount the shared folder to the local machine.
## Confirm if it can be mounted
$ showmount -e 157.20.20.2
## Mount
$ cd /mnt && mkdir media && mount -t nfs 157.20.20.2:/mnt/media /mnt/media
$ mkdir storage && mount -t nfs 157.20.20.2:/mnt/storage /mnt/storage
## Set fstab for automatic mounting at boot as mentioned earlier
Deploy qbittorrent using Docker.
docker run -d \
--name=qbittorrent \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-e WEBUI_PORT=8080 \
-p 8080:8080 \
-p 49152:49152 \
-p 49152:49152/udp \
-v /mnt/storage/qb-config:/config \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/qbittorrent:latest
Key points to note in the above code:
- The default port is 6881, which generally does not need to be changed. Most PT sites ban port 6881, so it can be changed to 49152 or others.
- The path set with -v must have the corresponding folder created before using the command for deployment.
- Replace the path
/mnt/storage/qb-config
in-v /mnt/storage/qb-config:/config
with the desired qbittorrent configuration path. All modifications will be saved, and if the container crashes, as long as the configuration is not lost, restarting the container will suffice. - The
/mnt/media/qb-downloads
part in-v /mnt/media/qb-downloads:/downloads
also needs to be modified.
Tip
The key points during the docker deployment process are the ports and mount paths. Just find the image on docker hub and modify it accordingly.
The subsequent text will not elaborate on the modification methods during the docker deployment process; the basic points are the same as those in this section.
K8s Deployment of qbittorrent#
Deploy qbittorrent through the kuboard panel.
Find a suitable namespace (you can manually create one) to create the workload.
Configure container information.
Mount storage.
Publish the application.
After saving, wait for the container log to indicate success, and you can access it through the defined published port. For example, in the image above, port 30001 is mapped to the container's 8080 port, so you can access it via http://157.20.20.3:30001. The default account for qb is admin/adminadmin.
Basic Settings for qb#
In the upper navigation bar of the qb page, there is a small gear icon, which is the entry for qb settings.
- Regarding the Download option, in the save management section, you need to set the red box part shown in the image below. The first setting is related to sonarr and radarr; when creating hard links through sonarr and radarr, the folder will be modified. Setting it to manual allows you to take over the download folder settings. The second part is to extract and back up the downloaded file's torrent. Usually, downloaded media files do not come with torrent files, so this setting can back up the torrent to a specified folder.
- In the BitTorrent tab, if you are searching for resources on sites like EZTV, RARGB, ACG.RIP, etc., checking "Enable DHT (decentralized network) to find more users" will yield better results. If you are using PT sites, you can uncheck this option.
The role of the tracker server is to help users increase download speed and search for more user nodes. Tracker servers can be copied and pasted from trackerslist.com.
- In the Advanced tab, checking the two tracker-related options helps connect to more tracker servers, thereby improving speed.
Scraping#
Regarding the scraping method, this plan uses sonarr and radarr to create hard links for downloaded resource files. The principle is to rename the files so that media servers like Emby and Plex can connect to TMDB, IMDB, and other internet movie databases to search for corresponding information, thus displaying posters, summaries, actor information, etc., in the media library.
Among them, sonarr corresponds to scraping TV series, while radarr corresponds to scraping movies. Sonarr and radarr can index public BT sites through jackett to achieve local searches for internet series, enabling qb to automatically download and scrape.
Expand
Deployment#
This section only introduces the deployment method using Docker. The deployment methods and notes for jackett, sonarr, and radarr are the same as the qb deployment method mentioned earlier. The k8s deployment method refers to the qb deployment method.
Tip
Remember to create the corresponding folders.
Jackett#
docker run -d \
--name=jackett \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-e AUTO_UPDATE=true \
-e RUN_OPTS= \
-p 9117:9117 \
-v /mnt/storage/jackett-config:/config \
-v /mnt/media:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/jackett:latest
Sonarr#
docker run -d \
--name=sonarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 8989:8989 \
-v /mnt/storage/sonarr-config:/config \
-v /mnt/media/tv:/tv \
-v /mnt/media/anime:/anime \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/sonarr:latest
Radarr#
docker run -d \
--name=radarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 7878:7878 \
-v //mnt/media/radarr-config:/config \
-v /mnt/media/movies:/movies \
-v /mnt/media/qb-downloads:/downloads \
--restart unless-stopped \
lscr.io/linuxserver/radarr:latest
Overseerr#
docker run -d \
--name=overseerr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 5055:5055 \
-v /mnt/storage/overseerr-config:/config \
--restart unless-stopped \
lscr.io/linuxserver/overseerr:latest
Settings#
Jackett#
Access jackett via http://ip:9117 and click Add indexer to add.
Sonarr#
Access sonarr via http://ip:8989.
In settings --> media management, set the media folder monitoring. Sonarr will automatically create hard links from the downloaded series folder to this location.
In settings --> profiles, set the default language.
In index, set the default indexer, which requires the API key from jackett. The added subscription index format should be Torznab.
In settings --> connect, set the media center and obtain the API key from Emby. This part can wait until Emby is deployed.
In settings --> download clients, set the connection to the qb downloader and create the corresponding monitoring directory.
In settings --> metadata, set the metadata information to be retrieved.
Once the above settings are completed, you can set up searches and monitoring for series in the Series section. It will also notify qb to download, and after downloading, it will automatically hard link to the target folder for scraping.
Radarr#
Radarr's basic settings are similar to Sonarr, but Radarr can be set to Chinese.
Log in via http://ip:7878, and in settings --> UI, you can set the page to Chinese.
In media management, adding "" can set the renaming to Chinese directories.
Overseerr#
Overseerr is a beautiful and powerful media management tool, which can be considered a combination of radarr and sonarr. It manages sonarr and radarr through API keys. After deployment, you will enter a guided setup page, where the API keys used can be found in sonarr and radarr's settings --> general.
Important points to note:
- This software relies on the plex server, so please deploy the plex server first.
- In general settings, select the display language as Chinese and the region as all.
- TV and anime can be added separately in the server.
- Media scraping heavily relies on TMDB access, so please set up a global proxy. For example, if using a soft router for proxy access, if the proxy is set within the virtual machine where overseerr is located, please add
-e https_proxy=http://172.17.0.1:7890
during the docker deployment process; otherwise, overseerr will not be able to set the proxy.
Media Library (Emby and Plex)#
Expand
Refer to the deployment methods for emby and plex on DockerHub for reference. Here are examples:
Emby#
docker run -d \
--name=emby \
-e UID=1000 \
-e GID=1000 \
-e TZ=Asia/Shanghai \
-p 8096:8096 \
-p 8920:8920 \
-v /mnt/storage/emby-config:/config \
-v /mnt/storage/emby-lib:/lib \
-v /mnt/media:/media \
--restart unless-stopped \
lscr.io/linuxserver/emby:latest
Plex#
docker run -d \
--name plex \
--network=host \
--restart=always \
-p 32400:32400/tcp \
-p 8324:8324/tcp \
-p 32469:32469/tcp \
-p 1900:1900/udp \
-p 32410:32410/udp \
-p 32412:32412/udp \
-p 32413:32413/udp \
-p 32414:32414/udp \
-e TZ="Asia/Shanghai" \
-e PLEX_CLAIM="plex_claim_code" \
-e ADVERTISE_IP="http://ip:32400/" \
-e PUID=1000 \
-e PGID=1000 \
-v /mnt/storage/plex/database:/config \
-v /mnt/storage/plex/transcode:/transcode \
-v /mnt/media:/data \
plexinc/pms-docker
The plex_claim can be obtained from the page: https://www.plex.tv/zh/claim/
Tip
Plex is not open-source media library software; its account management requires registration on the official website, and the plex_claim_code is obtained through the account.
Subtitle Section#
During the resource downloading process, some films may not have embedded subtitles, making it difficult to understand foreign languages. Configuring a subtitle server can solve this problem.
Expand
Deployment#
docker run -d \
--name=bazarr \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Shanghai \
-p 6767:6767 \
-v /mnt/storage/bazarr-config:/config \
-v /mnt/media/movies:/movies \
-v /mnt/media/tv:/tv \
-v /mnt/media/anime:/anime \
--restart unless-stopped \
lscr.io/linuxserver/bazarr:latest
Settings#
The general settings in bazarr are basically consistent with those in radarr and sonarr.
The difference is that there are three places where the API key needs to be set. The API keys for sonarr and radarr can be obtained from their respective pages, while the subtitle download API key needs to be obtained by registering an account on the corresponding subtitle website. I am using Assrt.net (pseudo), with the registration link provided in the text.