ISP Backbone Traffic in Japan

updated: August 2024 by Kenjiro Cho

Overview

We are collecting month-long aggregated traffic logs for different traffic groups from 9 major ISPs in Japan twice per year, in May and November, in order to analyze the macro-level impact of residential broadband traffic. These traffic groups are carefully selected to be summable, and not to count the same traffic multiple times.

The participating ISPs are BIGLOBE, IIJ, J:COM, KDDI, Nifty, NTT Communications, NTT Plala, Optage, and SoftBank (former Softbank Telecom and fomer Softbank BB).

Brief History of the Study

• 2000-2005: e-Japan strategy (by IT strategic headquarter) was successful in broadband deployment.
• 2004/06: next generation IP intrastructure report by MIC, an output of one of governmental study groups, identified issues in backbone: (1)emphasized importance of long-term measurements for policy making, (2)suggested cooperative measurement by IPSs, academia, and government.
• 2004/07: a study group was formed with 7 ISPs, 4 researchers from academia, MIC as secretariat. First, the group had technical discussions among ops/research people, and then, talked to top management to start cooperative data collection.
• 2010/11: data from POWEREDCOM is merged into that of KDDI as a result of the network integration after the merger of the 2 ISPs.
• 2011/05: BBIX and Equinix joined the study groutp as IX partners.
• 2015/05: Softbank Telecom and Softbank BB are merged into Softbank, but the data is still provided separately.
• 2017/05: 4 new ISPs, BIGLOBE, J:COM, Nifty and NTT Plala joined the group, but their data is not merged until the consistency is confirmed.
• 2019/05: The 4 new ISPs' data is merged into the offical release, and the past data from 2017/05 is also updated with the merged data.
• This is not an official governmental activity but ISPs' voluntary actions partly because ISPs are concerned about government intervention.

Background

Fig. 1 shows the number of boradband subscriners in Japan. (The gap at 2010/03 is due to a change in data collection methods of some CATV service providers. The cap at 2023/06 is due to an amended contract number from an ISP.)

Fig. 2 shows aggregated traffic and its growth rate at major IXes in Japan, namely JPIX, JPNAP, NSPIXP, BBIX, and Equinix.

Methodology

We have been collecting traffic volumes crossing ISP boundaries which can be roughly divided into customer traffic, and external provider traffic such as peering and transit. For practical purposes, we selected the five traffic groups in for data collection:

(A1) RBB customers

represent residential broadband customer lines. This group also includes small business customers using residential broadband access.

(A2) non-RBB customers

represent customer lines other than RBB customers, including leased lines, data centers, and dialup lines. This group includes RBB customers behind leased lines, e.g., second or third level ISPs, since ISPs do not distinguish them from other leased lines. Customer providers, back offices, and CDN caches are also included.

(B1) external 6IXes

represent links for 6 major IXes, namely JPIX, JPNAP and NSPIXP in both Tokyo and Osaka in order to compare measurements at these IXes as well as to know the traffic share of our measurement.

(B2) external domestic

represents domestic external provider links other than the 6IXes, including regional IXes, private peering and transit. We used the term ``domestic'' to indicate that both ends of a link are located in Japan. This group also includes domestic peering with global ASes.

(B3) external international

represents international external provider links with one end point outside of Japan.

These groups are chosen by the existing operational practices of the participating ISPs because it is impossible to draw a strict line for grouping, e.g., residential/business and domestic/international, on the global Internet. We re-aggregate each ISP's aggregated logs, and only the resulting aggregated traffic is used in our study so as to not reveal the share of each ISP.

Each ISP provided month-long traffic logs aggregated for each traffic group by a log aggregation tool we developed. This allows ISPs not to disclose the internal structure of their network or unneeded details of their traffic. The final results were obtained by aggregating all traffic logs provided by the six ISPs.

The time resolution of the logs is 2 hours since it was the highest common factor for month-long data in MRTG and RRDtool; 2-hour boundaries in UTC fall on odd hours in Japanese Standard Time (UTC+9). Note that, IN and OUT are presented from the ISPs' point of view.

Traffic measured by the 9 ISPs

Customer Traffic

External Traffic

Traffic Growth

The monthly average rates in bits/second of the traffic groups are shown in Table 2 and their growth is illustrated in Fig. 9 and 10.

The decrease at 2010/05 seems to be an impact of the amended Copyright Act making the download of copyright infringing content illegal. The increase of A2 IN at 2016/11 is a result of re-examination for classifying CDN caches into A2. The gap between 2016/11 and 2017/05 is a transition from 5 ISPs to 9 ISPs.

Annual growth rates

The annual growth rates over the years are summarized in Table 2. The growth rate is the ratio against the measurement one year before.

Estimating Total Customer Traffic in Japan

We can estimate the total customer traffic in Japan using the measurements from the 9 ISPs. We use the participating ISPs' share to compute the total traffic in Japan. For A1, the contract share of broadband services of the participating ISPs are used (assuming that customers traffic usage is similar among ISPs). For A2, the traffic share of the major IXes is used. However, the A2 data is provided only from 4 ISPs, and the A2 ratio differs from ISP to ISP. So, the number for A2 is just a very rough estimate.

References

1. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
   The Impact and Implications of the Growth in Residential User-to-User Traffic.
   SIGCOMM2006, pp207-218. Pisa, Italy. September 2006. (pdf).
2. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
   Observing Slow Crustal Movement in Residential User Traffic.
   ACM CoNEXT2008. Madrid, Spain. December 2008. (pdf).