Deploying a local apt source on Raspberry Pi

Preface#

The Raspberry Pi officially joins the internal network environment, preparing to deploy the apt source to provide source repository services for other Debian-based devices or virtual machines in the internal network environment, such as Proxmox. If you want to simulate an enterprise data center environment, it is necessary to completely isolate the internal and external networks.

Understanding Local Sources#

A local source refers to the source of software packages. Local source devices provide source services for devices that cannot connect to the internet in the internal network environment. Why can't they connect to the internet? In data centers or other internal network environments, some devices cannot connect to the internet for security reasons; it's not that they cannot, but that they are not allowed to connect, with a focus on the word "security." Therefore, the local source becomes the only access point for devices in the internal network environment to update software packages. Thus, it is necessary to download the source from the internet to the device storing the local source. For example, I installed Proxmox on my server, which is a virtual machine system based on the Debian operating system, and it will serve as the main system for internal network services in the future, virtualizing multiple devices to provide different services. However, after installation, it has various software missing issues and cannot connect to the internet for installation, so a local source device is needed to meet its software installation needs.

How to Deploy a Local Source#

First, there are tutorials online that can be referenced for deployment issues. The Raspberry Pi system is uninstalled from the TF card, and it is generally 32G when purchased. With this capacity, let alone deploying a local source, just putting a few system image packages on it will fill it up. Therefore, we need to add a large-capacity backpack to the Raspberry Pi so that it can take on this task. Before deploying the local source, it is necessary to mount a mobile hard drive to the Raspberry Pi. I happen to have a 512G solid-state drive that can be used as its external disk.

# Find the disk
root@raspberrypi:/#fdisk -l
# Result omitted
Disk identifier: 0x6aef5cb3

Device         Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 62333951 61801472 29.5G 83 Linux


Disk /dev/sda: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: Generic
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt

Disk identifier: 8C903F36-23B9-44CB-99B8-D122B713E560Device     Start        End    Sectors   Size Type
/dev/sda1   2048 1000215182 1000213135 476.9G Linux filesystem

I found a disk sda, which was previously used with the Linux file system. First, clear the data and reformat it.

root@raspberrypi:/# mkfs -t ext3 /dev/sda
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 125026902 4k blocks and 31260672 inodes
Filesystem UUID: 2898dbe1-7892-4424-9fee-ff8b78b8b32e
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Create a mount directory and mount the disk.

root@raspberrypi:/#mkdir /opt/sda1
root@raspberrypi:/#chmod 777 /opt/sda1
root@raspberrypi:/#mount /dev/sda /opt/sda1

Query the disk's UUID.

root@raspberrypi:/# blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="0F92-BECC" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="6aef5cb3-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="41c98998-6a08-4389-bf74-79c9efcf0739" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6aef5cb3-02"
/dev/sda: UUID="2898dbe1-7892-4424-9fee-ff8b78b8b32e" BLOCK_SIZE="4096" TYPE="ext3"

Set the disk to mount at boot, as it would be troublesome if it fails to mount after a crash and reboot, and you don't know where the problem is.

root@raspberrypi:/etc# vim fstab
# Add the following content to the last line of fstab, with the UUID obtained in the previous step
UUID=2898dbe1-7892-4424-9fee-ff8b78b8b32e /opt/sda1 ext3 defaults        0       0

The disk is mounted, and we can start deploying the apt local source. Overall, there are usually two ways to set up a local source:

On-demand pulling, where internal network devices install packages through remote copying as needed. References 1, 2, and 3 in the materials are all this method.
Directly deploy a local repository, pulling the majority of software packages. Reference 4 in the materials is this method. Since we have prepared a large-capacity hard drive, we are definitely considering the latter. Moreover, since it mainly serves the Debian-based Proxmox system, we need to replace the source target during the deployment process.

root@raspberrypi:/# apt install apt-mirror
root@raspberrypi:/# cd /etc/apt
root@raspberrypi:/etc/apt# vim mirror.list
# Modify the mirror.list file
# After modification, execute apt-mirror
root@raspberrypi:/# apt-mirror

The content of the file is as follows, changing the original official US source to the Chinese source.

############# config ##################
#
set base_path    /opt/sda1/pve_mirror
#
#set mirror_path   $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb http://ftp.cn.debian.org/debian unstable main contrib non-free
deb-src http://ftp.cn.debian.org/debian unstable main contrib non-free

# mirror additional architectures
#deb-alpha http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-amd64 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-armel http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-hppa http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-i386 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-ia64 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-m68k http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-mips http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-mipsel http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-powerpc http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-s390 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-sparc http://ftp.us.debian.org/debian unstable main contrib non-free

clean http://ftp.us.debian.org/debian

After executing apt-mirror, it will take a very long time. The first time I thought it failed and terminated it with ctrl+C. Upon executing it again, I found that what originally required 102G had become 92G, indicating that although it was slow, it was indeed downloading.

After spending a day, it finally finished downloading. Next, we need to deploy an Apache proxy for internal network devices to access.

root@raspberrypi:/#apt-get install apache2
root@raspberrypi:/#ln -s /opt/sda1/pve_mirror/mirror/ftp.cn.debian.org/debian /var/www/html/proxmox

After creating the symbolic link, you can access the software source on this Raspberry Pi through a browser. The access address is http://ip/pve.

Of course, there are other methods. If configuring apache2 or nginx doesn't work, there are still other methods. Installing Node.js and using an open-source file service system to build a link is also feasible, such as hope-server on GitHub.

root@raspberrypi:/#apt install nodejs
root@raspberrypi:/#apt install npm
root@raspberrypi:/#apt install yarn
root@raspberrypi:/#yarn global install hope-server
root@raspberrypi:/#hope-server -d <filepath> -p 8080 -o localhost
# filepath is the software source directory, which can be the disk mount path or the soft link path after mounting the disk.
# However, this method has certain drawbacks; when starting hope-server, it will occupy the console, so you can use nohup + crontab.
root@raspberrypi:~# nohup hope-server -d /var/www/proxmox/ -p 8080 -o localhost &
[3] 6843
---------------------------------------------------
root@raspberrypi:~# crontab -e
no crontab for root - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

Choose 1-4 [1]: 2
# Add the following line at the end of the editor
@reboot nohup hope-server -d /var/www/proxmox/ -p 8080 -o localhost &
# This way, it will execute every time it restarts.

How to Modify the PVE Source#

After deploying the local source, the next step is to modify the source address on the client, which is the PVE system in the internal network, with the path being /etc/apt/sources.list.

root@proxmox:/#nano /etc/apt/sources.list
# Add a source address similar to the one below in the editor
deb [trusted=yes] http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian bullseye main contrib

deb [trusted=yes] http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian bullseye-updates main contrib

deb [trusted=yes] http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian bullseye-backports main contrib

deb [trusted=yes] http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian bullseye-security main contrib

Although it is already considered successful at this point, it still unexpectedly reported an error. 😅

Reading package lists... Done
E: Release file for http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian/dists/bullseye/InRelease is not valid yet (invalid for another 33d 17h 53min 33s). Updates for this repository will not be applied.
E: Release file for http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian/dists/bullseye-updates/InRelease is not valid yet (invalid for another 130d 21h 52min 40s). Updates for this repository will not be applied.
E: Release file for http://172.20.0.2:80/proxmox/mirror/mirrors.tuna.tsinghua.edu.cn/debian/dists/bullseye-backports/InRelease is not valid yet (invalid for another 130d 21h 52min 40s). Updates for this repository will not be applied.

It seems to be a time issue; perhaps the timezone selected during the installation of the PVE system was incorrect? Let's check the current time of the PVE system.

root@proxmox:~# date
Mon 08 Aug 2022 12:29:02 AM CST

Modify it.

root@proxmox:/etc# vim /etc/systemd/timesyncd.conf
# Set the NTP address to the soft router
root@proxmox:/etc# systemctl restart systemd-timesyncd.service
root@proxmox:/etc# systemctl enable systemd-timesyncd.service
root@proxmox:/etc# date
Sat 17 Dec 2022 07:21:07 AM CST

Finally, apt update can be performed normally.

Aki