Mirroring at Savannah
NOTE: This is an updated version of the previous Mirmon page.
Savannah offers two public-facing mirroring services:
ftpmirror.gnu.org
- redirects to world-wide mirrors offtp.gnu.org
. These mirrors carry official releases of GNU pacakges. This page refers to it asgnu mirror
.download.savannah.{nongnu,gnu}.org
- redirects to world-wide mirrors of Savannah download area. Typically used for non-GNU software, but GNU packages can store files there as well. This page refers to it asnongnu mirror
.
Adding mirrors
Occasionally, a new mirror is submitted. Usually the webmasters inform us, but people may come to us directly.
When we are told of a new mirror, we verify it, then
- On mgt1, add contact information to
/root/administration/mirrors-contacts.txt
; - On download0.savannah.gnu.org, add the mirror URLs to
/srv/download/00_MIRRORS.html
(alphabetical, check list online) and/srv/download/00_MIRRORS.txt
(no order).
Client-side overview
A user requesting a download from ftpmirror.gnu.org
will receive
an HTTP/302 response redirecting to a mirror closest to the user's
originating IP:
$ curl -o /dev/null -D /dev/stdout --silent http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
HTTP/1.1 302 Found
Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
[...]
A user requested a download from download.savannah.gnu.org
will be
redirected to an intermediate redirection URL, and then to the mirror:
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases/test-project/README
HTTP/1.1 302 Found
Location: http://download.savannah.gnu.org/releases-redirect/test-project/README
[...]
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases-redirect/test-project/README
HTTP/1.1 302 Found
Location: http://nongnu.askapache.com//test-project/README
[...]
The technical reason for this extra redirection is that
download.savannah.gnu.org
serves directory listing directly, and
only redirects actual files. See nginx details below.
Server-side redirection overview
Redirection is implemented using a Perl CGI script,
called from nginx running on download0.sv.gnu.org
.
Both gnu and nongnu redirections are served from
download0.savannah.gnu.org VM. ftpmirror.gnu.org
is
CNAME'd to ftpmirror.sv.gnu.org
, which points
to the same IP as download0 (remember Savannah admins
only control *.sv.gnu.org DNS entries).
The nginx configuration files are:
download0:/etc/nginx/sites-available/ftpmirror
,
download0:/etc/nginx/sites-available/ftpmirror-common.inc
,
download0:/etc/nginx/sites-available/ftpmirror-common-https.inc
,
download0:/etc/nginx/sites-available/download
,
download0:/etc/nginx/sites-available/download-common.inc
,
download0:/etc/nginx/sites-available/download-common-https.inc
.
The *-common*.inc
files contain the CGI configurations.
The other files include them twice (once for HTTP, once for HTTPS).
The HTTPS CGI configuration for gnu mirror
is:
location / {
gzip off;
# In this case, the ENTIRE path is passed on the to cgi script, e.g.
# given http://ftpmirror.sv.gnu.org/datamash/datamash-1.1.0.tar.gz we pass
# '/datamash/datamash-1.1.0.tar.gz' to the CGI script.
# So there is no need to use 'fastcgi_split_path_info' command.
fastcgi_param PATH_INFO $uri;
fastcgi_param MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/gnu.txt;
fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The CGI configuration for nongnu mirror
is:
# Redirect to the CGI if it's a file URL.
rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;
# Directory listing if it's a directory.
location /releases {
alias /srv/download;
fancyindex on; # Enable fancy indexes.
}
# Actual redirection mechanism.
location /releases-redirect {
gzip off;
# This command tells NGINX how to extract the path info part
# from the requested URL: it trims the 'releases-redirect' part,
# leaving only what follows it.
fastcgi_split_path_info ^(/releases-redirect)(/?.+)$;
# This command takes NGINX's path_info variable (extracted above),
# and sends it to the FastCGI daemon as the PATH_INFO environment variable.
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/download.txt;
fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The HTTPS versions substitute gnu.txt and download.txt with gnu-https.txt and download-https.txt
The mirror list text files (gnu.txt
and download.txt
)
contain an auto-generated list of active mirrors and region redirections:
# head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
http://us.mirrors.cicku.me/gnu/ us
http://mirror.freedif.org/GNU/ sg
http://fosszone.csd.auth.gr/gnu/ gr
http://mirror.marwan.ma/gnu/ ma
http://ftp.task.gda.pl/pub/gnu/ pl
# tail -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
ye az
yt za
za za
zm za
zw za
(Yemen [YE] uses the mirrors located in Azerbaijan [AZ], Zimbabwe [ZW] uses the mirrors located in South Africa [ZA].)
See 'mirmon' section below to learn how these files are generated.
The Perl script mirror-redirect uses Geo::IP module to detect users' regions based on their IP.
Server-side Mirmon overview
NOTE: Most of this section was copied from the Mirmon page and updated as needed.
The authoritative lists of mirrors are managed manually. For
gnu mirror
, GNU webmasters update the lists (http://www.gnu.org/prep/ftp.html, generated from http://www.gnu.org/prep/FTP). Fornongnu mirror
, Savannah admins maintain the lists as explained above.mirmon is used to check each mirror and determine how up-to-date it is.
We maintain 3 separate mirmon configurations:
gnu
- list of HTTP-only mirrors for ftpmirror.gnu.org. Thegnu
list is used later by the redirection CGI script.allgnu
- list of HTTP, FTP, RSYNC mirrors for ftpmirror.gnu.org. Theallgnu
list is used only by human admins to check which mirrors are up-to-date.nongnu
- list of HTTP-only mirrors for download.savannah.gnu.org. Thenongnu
list is used later by the redirection CGI script.
The mirmon input lists (of mirrors to check) are here:
The lists are stored here:
$ ls -lhog /opt/savannah/mirrors/mirmon-lists/ -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt lrwxrwxrwx 1 28 Feb 20 23:28 nongnu-mirror-list.txt -> /srv/download/00_MIRRORS.txt
The script sv_gnu-mirmon fetches the gnu list and saves it locally in the two versions (
gnu
andallgnu
). Thenongnu
list is symlinked to the manually-managed file.These files are the input to mirmon.
The three mirmon configuration files are:
$ ls -1 /etc/mirmon/ mirmon-allgnu.conf mirmon-gnu.conf mirmon-nongnu.conf
When
mirmon
is executed (once for each configuration), it reads the corresponding list file, and generates an HTML report page and a 'state' textual file:/opt/savannah/www/mirmon/gnu/index.html /opt/savannah/www/mirmon/nongnu/index.html /opt/savannah/www/mirmon/allgnu/index.html # ls -lhog /var/lib/mirmon/state-* -rw-r--r-- 1 16K Feb 21 19:42 /var/lib/mirmon/state-allgnu -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
A 'state' file contains the status for each mirror:
$ head -n5 /var/lib/mirmon/state-gnu http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.gnu.org/ 1707624961 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721 http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1707617761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121 http://ftp.cc.uoc.gr/mirrors/gnu/ 1707581761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121 http://ftp.eq.uc.pt/software/unix/gnu/ 1707621361 ok 1707657721 ssssssssssssfs 1707628921-ssssssssssssss 1707657721 http://ftp.fau.de/gnu/ 1707646561 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721
Two auxiliary scripts
/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh
trivially invoke sv_mirmon-to-geip; that script reads the state file and the mirror list and generates simple text files containing the active and up-to-date mirrors and region redirections:# ls /opt/savannah/mirrors/active-mirror-lists/ download-https.txt download.txt gnu-https.txt gnu.txt
These are the four files used in the CGI scripts (see above section).
The 3 mirmon invocations are executed as cronjobs in
/etc/cron.d/sv-mirmon
:# Update the authoritative list of GNU ftp mirrors (mirroring ftp.gnu.org). 3 1 * * * root sv_gnu-mirmon --http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt 6 1 * * * root sv_gnu-mirmon --all /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt # Check the status of each mirror (and regenarate the HTML report page). # For gnu/nongnu - also update the list of active mirrors (for the mirror CGIs). 22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get update >/var/log/mirmon/gnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh 32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q -get update >/var/log/mirmon/nongnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh 42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q -get update >/var/log/mirmon/allgnu 2>&1
Diagnostics and Troubleshooting
The three mirmon HTML reports are viewable here: http://download.savannah.gnu.org/mirmon/.
An additional page is used for testing: http://download.savannah.gnu.org/mirror-check/, http://ftpmirror.savannah.gnu.org/mirror-check/, https://download.savannah.gnu.org/mirror-check/, https://ftpmirror.savannah.gnu.org/mirror-check/.
The NGINX configurations are
*-common.inc
:location /mirror-check/ { gzip off; fastcgi_split_path_info ^(/mirror-check)(/?.+)$; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_param MIRROR_FILE /path/to/active/mirror/list; fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect; fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat; fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat; fastcgi_param TEST 1; include fastcgi_params; fastcgi_pass unix:/var/run/fcgiwrap.socket; }
Mirmon cronjob logs are saved in
/var/log/mirmon
. Current implementation saves only the most recent run.Mirmon state files should be generated at 22/32/42 minutes of each hour (defined in the cron file):
# ls -lhog /var/lib/mirmon/ total 84K -rw-r--r-- 1 27K Feb 11 09:17 debian-mirror-check.html -rw-r--r-- 1 3.9K Feb 11 09:17 state -rw-r--r-- 1 24K Feb 11 08:42 state-allgnu -rw-r--r-- 1 18K Feb 11 08:22 state-gnu -rw-r--r-- 1 4.2K Feb 11 08:32 state-nongnu
The active mirror list files should have similar timestamp:
# ls -lhog /opt/savannah/mirrors/active-mirror-lists/ total 24K -rw-r--r-- 1 1.7K Feb 11 08:32 download-https.txt -rw-r--r-- 1 2.1K Feb 11 08:32 download.txt -rw-r--r-- 1 4.4K Feb 11 08:22 gnu-https.txt -rw-r--r-- 1 4.2K Feb 11 08:22 gnu.txt