ISP Backbone Traffic in Japan

updated: August 2024 by Kenjiro Cho


Overview

We are collecting month-long aggregated traffic logs for different traffic groups from 9 major ISPs in Japan twice per year, in May and November, in order to analyze the macro-level impact of residential broadband traffic. These traffic groups are carefully selected to be summable, and not to count the same traffic multiple times.

The participating ISPs are BIGLOBE, IIJ, J:COM, KDDI, Nifty, NTT Communications, NTT Plala, Optage, and SoftBank (former Softbank Telecom and fomer Softbank BB).

Brief History of the Study

Background

Fig. 1 shows the number of boradband subscriners in Japan. (The gap at 2010/03 is due to a change in data collection methods of some CATV service providers. The cap at 2023/06 is due to an amended contract number from an ISP.)

broadband subscribers in Japan
Fig. 1: Broadband Subscribers in Japan

Fig. 2 shows aggregated traffic and its growth rate at major IXes in Japan, namely JPIX, JPNAP, NSPIXP, BBIX, and Equinix.

traffic at major IXes
Fig. 2: Traffic at Major IXes

Methodology

We have been collecting traffic volumes crossing ISP boundaries which can be roughly divided into customer traffic, and external provider traffic such as peering and transit. For practical purposes, we selected the five traffic groups in for data collection:

(A1) RBB customers
represent residential broadband customer lines. This group also includes small business customers using residential broadband access.
(A2) non-RBB customers
represent customer lines other than RBB customers, including leased lines, data centers, and dialup lines. This group includes RBB customers behind leased lines, e.g., second or third level ISPs, since ISPs do not distinguish them from other leased lines. Customer providers, back offices, and CDN caches are also included.
(B1) external 6IXes
represent links for 6 major IXes, namely JPIX, JPNAP and NSPIXP in both Tokyo and Osaka in order to compare measurements at these IXes as well as to know the traffic share of our measurement.
(B2) external domestic
represents domestic external provider links other than the 6IXes, including regional IXes, private peering and transit. We used the term ``domestic'' to indicate that both ends of a link are located in Japan. This group also includes domestic peering with global ASes.
(B3) external international
represents international external provider links with one end point outside of Japan.
figure of traffic groups
Fig. 3: Five traffic groups for data collection at customer and external boundaries of an ISP

These groups are chosen by the existing operational practices of the participating ISPs because it is impossible to draw a strict line for grouping, e.g., residential/business and domestic/international, on the global Internet. We re-aggregate each ISP's aggregated logs, and only the resulting aggregated traffic is used in our study so as to not reveal the share of each ISP.

Each ISP provided month-long traffic logs aggregated for each traffic group by a log aggregation tool we developed. This allows ISPs not to disclose the internal structure of their network or unneeded details of their traffic. The final results were obtained by aggregating all traffic logs provided by the six ISPs.

The time resolution of the logs is 2 hours since it was the highest common factor for month-long data in MRTG and RRDtool; 2-hour boundaries in UTC fall on odd hours in Japanese Standard Time (UTC+9). Note that, IN and OUT are presented from the ISPs' point of view.

Traffic measured by the 9 ISPs

For weekly data analysis, we took the averages of the same weekdays in a month but excluded holidays since holiday traffic patterns are closer to weekends.

Customer Traffic

week plot of customer-bb
Fig. 4: Week View of (A1) Customer Broadband
week plot of customer-other
Fig. 5: Week View of (A2) Customer Other

External Traffic

week plot of external-6ix
Fig. 6: Week View of (B1) External 6IX
week plot of external-domestic
Fig. 7: Week View of (B2) External Domestic
week plot of external-international
Fig. 8: Week View of (B1) External International

Traffic Growth

The monthly average rates in bits/second of the traffic groups are shown in Table 2 and their growth is illustrated in Fig. 9 and 10.

growth of customer traffic
Fig. 9: Growth of Customer Traffic
growth of external traffic
Fig. 10: Growth of External Traffic

The decrease at 2010/05 seems to be an impact of the amended Copyright Act making the download of copyright infringing content illegal. The increase of A2 IN at 2016/11 is a result of re-examination for classifying CDN caches into A2. The gap between 2016/11 and 2017/05 is a transition from 5 ISPs to 9 ISPs.

Table. 1: Monthly average rates of aggregated customer and external traffic (bps)
yearmonth(A1)customer-bb(A2)customer-other (B1)external-6ix(B2)external-domestic(B3)external-international
in(Gbps)out(Gbps)in(Gbps)out(Gbps) in(Gbps)out(Gbps)in(Gbps)out(Gbps)in(Gbps)out(Gbps)
20040998.1111.814.013.635.930.948.237.825.314.1
10108.3124.915.014.936.331.853.141.627.715.4
11116.0133.016.215.638.033.055.143.328.516.7
20055134.5178.323.723.947.941.673.358.440.124.1
11146.7194.236.129.754.048.180.968.157.139.8
200605173.0226.242.938.366.260.194.977.668.547.8
11194.5264.250.746.768.462.3107.690.594.557.8
200705217.3306.073.857.877.470.8124.5108.4116.471.2
11237.2339.885.463.293.583.4129.0113.3133.781.8
200805269.0374.7107.085.095.788.3141.2119.4152.694.4
11302.0432.9122.488.7107.5102.5155.6132.3176.1110.8
200905349.5501.0154.4121.4111.7104.9185.0155.4213.1126.4
11373.6539.7169.4127.6114.3109.8209.5154.3248.2148.3
201005321.9536.4178.8131.294.191.0194.8121.4286.9155.5
11311.1593.0190.1147.590.191.6198.7117.2330.1144.9
201105302.5662.0193.9174.498.490.0242.9131.5420.9160.5
11293.6744.5221.9207.5102.989.4265.1139.1498.5169.6
201205287.8756.6251.5243.0118.498.6317.4145.1528.7178.8
11294.0840.3268.3257.2103.283.2316.6135.7571.3201.6
201305347.81027.8300.3286.4114.585.5423.3161.3633.9231.6
11370.01146.3336.5326.2138.994.9520.8186.2714.5259.7
201405398.91274.5359.2317.2163.6101.5614.9214.3808.3282.3
11407.61557.0496.1426.1192.3104.6765.1246.5924.6340.6
201505457.01928.9525.6440.2198.9117.5955.6287.5941.5308.1
11452.92336.1581.1503.0251.9137.11306.4366.61059.7307.9
201605551.52863.3652.7570.5277.0112.61765.1453.81080.1292.4
11602.53396.61246.0653.6311.0113.61989.2518.21221.9535.8
201705954.85452.91390.0597.19590.5179.13207.1685.2.91283.1322.6
11779.15980.21428.9688.1690.6157.13591.1661.61437.5362.5
201805870.16827.91441.9726.4736.8214.73864.7559.41746.4452.6
11929.17281.81921.4867.5964.9283.44848.6710.51669.2400.9
2019051016.77859.62159.4948.9950.2289.45519.1848.91671.0408.5
111073.08641.02323.4956.5994.1290.86232.5901.21995.5540.9
2020051534.312575.62968.12420.11610.7328.610065.51353.32945.8724.5
111542.712885.52787.32552.41502.0290.59380.01535.12603.5593.5
2021051776.415264.63226.43084.71881.8584.312454.51651.12946.1715.6
111772.314885.53590.73147.52078.7631.912906.81654.02518.9820.7
2022051922.116180.73850.43530.72299.0677.614178.91687.82492.9914.1
111973.217749.14039.43827.92616.8707.715662.51952.62687.0939.1
2023052043.218200.64295.44104.32889.1753.316016.72059.42860.3986.9
112135.619814.94514.44607.73546.1906.316830.32190.83154.6962.7
2024052255.820476.34862.94677.23945.3972.317052.82427.93011.41013.2
11

Annual growth rates

The annual growth rates over the years are summarized in Table 2. The growth rate is the ratio against the measurement one year before.

Table. 2: Annual Growth Rates
yearmonth(A1)customer-bb(A2)customer-other (B1)external-6ix(B2)external-domestic(B3)external-international
inoutinout inoutinoutinout
2005111.2651.4602.2281.9041.4211.4581.4681.5732.0042.383
2006051.2861.2691.8101.6031.3821.4451.2951.3291.7081.983
111.3261.3601.4041.5721.2671.2951.3301.3291.6551.452
2007051.2561.3531.7201.5091.1691.1781.3121.3971.6991.490
111.2201.2861.6841.3531.3671.3391.1991.2521.4151.415
2008051.2381.2251.4501.4711.2361.2471.1341.1011.3111.326
111.2731.2741.4331.4031.1501.2291.2061.1681.3171.355
2009051.2991.3371.4431.4281.1671.1881.3101.1751.3961.339
111.2371.2471.3841.4391.0631.0711.3461.1661.4091.338
2010050.9211.0711.1581.0810.8420.8681.0530.7811.3461.230
110.8331.0991.1221.1560.7880.8340.9480.7601.3300.977
2011050.9401.2341.0841.3290.9250.9551.3051.1091.4671.032
110.9441.2551.1671.4071.1420.9761.3341.1871.5101.170
2012050.9511.1431.2971.3931.2031.0961.3071.1031.2561.114
111.0011.1291.2091.2401.0030.9311.1940.9761.1461.189
2013051.2081.3581.1941.1790.9670.8671.3341.1121.1991.295
111.2591.3641.2541.2681.3461.1411.6451.3721.2511.288
2014051.1471.2401.1961.1081.4291.1871.4531.3291.2751.219
111.1021.3581.4741.3061.3841.1021.4691.3241.2941.312
2015051.1461.5131.4631.3881.2161.1581.5541.3421.1651.091
111.1111.5001.1711.1801.3101.3111.7071.4871.1460.904
2016051.2071.4841.2421.2961.3930.9581.8471.5781.1470.949
111.3301.4542.1441.2991.2350.8291.5231.4141.1531.149
2017051.3491.3801.9981.2111.5431.3011.5921.3791.1561.056
110.9911.2831.0811.1551.6541.0881.5541.1331.1380.986
2018050.9111.2541.0371.2171.2481.1991.2050.8161.3611.403
111.1931.2181.3451.2611.3971.8041.3501.0741.1611.106
2019051.1681.1491.4981.3061.2901.3481.4281.5180.9570.903
111.1551.1871.2091.1031.0301.0261.2851.2681.1951.349
2020051.5091.6001.3752.5501.6951.1351.8241.5941.7631.774
111.4381.4911.2002.6681.5110.9991.5051.7031.3051.097
2021051.1581.2141.0871.2751.1681.7781.2371.2201.0000.988
111.1491.1551.2881.2331.3842.1751.3761.0770.9681.383
2022051.0821.0601.1931.1451.2221.1601.1381.0220.8461.277
111.1131.1921.1251.2161.2591.1201.2141.1811.0671.144
2023051.0631.1251.1161.1621.2571.1121.1301.2201.1471.080
111.0821.1161.1181.2041.3551.2811.0751.1221.1741.025
2024051.1041.1251.1301.1401.3661.2911.0651.1791.0531.027
11

Estimating Total Customer Traffic in Japan

We can estimate the total customer traffic in Japan using the measurements from the 9 ISPs. We use the participating ISPs' share to compute the total traffic in Japan. For A1, the contract share of broadband services of the participating ISPs are used (assuming that customers traffic usage is similar among ISPs). For A2, the traffic share of the major IXes is used. However, the A2 data is provided only from 4 ISPs, and the A2 ratio differs from ISP to ISP. So, the number for A2 is just a very rough estimate.

estimated A1 total
Fig. 11: Estimated A1 Total in Japan. Mobile shows the total 3G/LTE data traffic in Japan.
estimated A2 total
Fig. 11: Estimated A2 Total in Japan
Table. 3: Estimated Total Customer Traffic in Japan
yearmonth6 ISPsestimated A1 total3 ISPsestimated A2 total
contract share(%)in(Gbps)out(Gbps)IX traffic share(%)in(Gbps)out(Gbps)
20040952.218821414.99491
1052.220823915.29998
1152.222225514.0116111
20050552.325734114.9159160
1150.129338715.9227187
20060549.734845516.7257229
1149.439453516.1315290
20070549.144362417.5422330
1148.449070216.6515381
20080547.356879217.9598475
1146.564993018.7655474
20090545.9762109017.4887698
1145.1828120017.6963725
20100543.8735122016.91060776
1143.9709135017.01120868
20110543.8691151013.814101260
1144.1666169012.817301620
20120544.1652171012.420301960
1144.3664190011.224002300
20130544.877622909.5631403000
1144.683025708.6738803760
20140544.190428908.7641003620
1143.793235607.1369605980
20150543.4105044507.3671405980
1142.7106054706.7985607410
20160541.9132068404.871340017700
1141.3146082304.532750014400
yearmonth9 ISPsestimated A1 total4 ISPsestimated A2 total
contract share(%)in(Gbps)out(Gbps)IX traffic share(%)in(Gbps)out(Gbps)
20170567.9137078406.801920010200
1167.2113086903.903660017600
20180566.51310103006.212360013700
1166.31400110006.013200014400
20190565.01560121006.183490015400
1168.31570126005.354340017900
20200566.12320190003.368830072000
1165.02330195002.26123000113000
20210563.92780239005.845520052800
1162.92820237005.10704000617000
20220562.33090260004.808020073600
1160.13280295004.399200087200
20230559.63430305004.2810000095900
1157.43720345004.44102000104000
20240556.83970360004.39111000107000
11

References

  1. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
    The Impact and Implications of the Growth in Residential User-to-User Traffic.
    SIGCOMM2006, pp207-218. Pisa, Italy. September 2006. (pdf).
  2. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
    Observing Slow Crustal Movement in Residential User Traffic.
    ACM CoNEXT2008. Madrid, Spain. December 2008. (pdf).