Sample Setup
Internet/ISP router
//
// +-----------------+ +---------+
// eth1| Linux firewall | | LAN |
//-----+ (Gateway) +------+ Servers |
| 192.168.1.254 |eth0 | Users |
+-----------------+ +---------+
- Router
- Traffic shaping device
- Firewall
- IP accounting at FORWARD chain
ISP port 1000MBps (Enterprise rack)
//
// +------------+ +---------+
// eth1| Linux LB0 | | Apache1 |
//-----+------------+------+---------+
| Linux LB1 |eth0 | Apache2 |
+------------+ +---------+
Firewall | Mysql1 |
123.1.2.1 +---------+
123.1.2.2 | Mysql2 |
123.1.2.3 +---------+
| memcache|
+---------+
- lb0 and lb1 - RHEL based Software load balancer. These two servers act as an IP failover device, firewall and reverse proxy software to speed up backends. IP accounting is configured at INPUT and OUTPUT chains for three vips (123.1.2.1, 123.1.2.2, and 123.1.2.3)
- 123.1.2.1, 123.1.2.2, and 123.1.2.3: Virtual IP address (VIPs) moves between lb0 and lb1 to provide service availability.
- Apache{1,2,3..N}, Mysql{1,2} : Actual servers for storing data, files and java applications.
WARNING! These examples may block your network access if not executed correctly. IP accounting requires a basic understanding of port, services, protocols, and networking concepts. Most of the actions listed in this FAQ are written with the assumption that they will be executed by the root user running the bash or any other modern shell. All examples are tested under RHEL or CentOS Linux; but should work under any other modern distro.iptables Based IP Accounting
# iptables -L -n To list all rules you use the above command. You need to pass the -v option, which will display additional information:
- Interface name (e.g., eth1).
- Rule options (if any).
- TOS masks.
- Packet and byte counters, with the suffix 'K', 'M' or 'G'.
# iptables -L -n -vSample outputs:
Chain INPUT (policy ACCEPT 73860 packets, 95M bytes)Please note the packets and bytes columns.
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 state RELATED,ESTABLISHED
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT 54182 packets, 5660K bytes)
pkts bytes target prot opt in out source destination
Step # 1: Create a User Defined Chain
The syntax is as follows to create a new user defined chain to set IP accounting:# iptables -N INET_OUT
# iptables -N INET_INThe above command created a new user-defined chain called INET_OUT to keep track of outgoing traffic. INET_IN can be used to keep track of incoming traffic.
Step #2: Setup IP Accounting
You need to attach INET_IN and INET_OUT as follows:# iptables -A FORWARD -j INET_IN
# iptables -A FORWARD -j INET_OUTThe FORWARD is used for packets being routed through the box i.e. your Linux server act as a router (see fig.01 Gateway IP Accounting). You can attach as follows for INPUT and OUTPUT chains:
# iptables -A INPUT -j INET_IN
# iptables -A OUTPUT -j INET_OUTStep #3: See IP Accounting Data
Type the following commands:# iptables -L -v -nTo see only outgoing IP accounting data:
# iptables -v -n -L INET_OUTSample outputs:
Chain INET_OUT (1 references)example.com generated 405k bytes and lan.local generated 16G outgoing traffic.
pkts bytes target prot opt in out source destination
2472 405K all -- any any example.com anywhere
17M 16G all -- any any lan.local anywhere
Task: Set IP Accounting By Address
Let us say you've three domains as follows and like to keep track of each domains outgoing and incoming traffic:- example.com - 123.1.2.1
- example.net - 123.1.2.2
- example.org - 123.1.2.3
# /sbin/iptables -N INET_OUT
# /sbin/iptables -N INET_IN
# /sbin/iptables -A INPUT -j INET_IN
# /sbin/iptables -A OUTPUT -j INET_OUT
# /sbin/iptables -A INET_IN -d 123.1.2.1
# /sbin/iptables -A INET_IN -d 123.1.2.2
# /sbin/iptables -A INET_IN -d 123.1.2.3
# /sbin/iptables -A INET_OUT -s 123.1.2.1
# /sbin/iptables -A INET_OUT -s 123.1.2.2
# /sbin/iptables -A INET_OUT -s 123.1.2.3
# /sbin/service iptables saveTo see incoming IP stats, enter:
# /sbin/iptables -v -n -L INET_INSample outputs:
Chain INET_IN (1 references)To see outgoing IP stats, enter:
pkts bytes target prot opt in out source destination
16M 2086M all -- any any anywhere example.com
6141K 1319M all -- any any anywhere www.example.net
30M 4657M all -- any any anywhere example.org
# /sbin/iptables -v -n -L INET_OUTSample outputs:
Chain INET_OUT (1 references)
pkts bytes target prot opt in out source destination
2472 405K all -- any any example.com anywhere
17M 16G all -- any any www.example.net anywhere
5165K 2399M all -- any any example.org anywhere
Task: IP Accounting By Service Port
To collect port 80 (http), 443 (https), 22 (SSH), 143 (imap), 25 (SMTP), 53 (dns) volume statistics for data carried on eth1 link, enter:### create a new set of chains ###To view stats, enter:
# /sbin/iptables -N INET_IN_PORT
# /sbin/iptables -N INET_OUT_PORT
### ATTACH them (USE FORWARD for gateway systems) ###
# /sbin/iptables -A INPUT -j INET_IN_PORT
# /sbin/iptables -A OUTPUT -j INET_OUT_PORT
### Set up rules for INPUT ###
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 80
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 443
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 25
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 143
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 53
# /sbin/iptables -A INET_IN_PORT -i eth1 -p udp --dport 53
# /sbin/iptables -A INET_IN_PORT -i eth1 -p tcp --dport 22
### Set up rules for OUTPUT ###
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 80
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 443
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 25
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 143
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 53
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p udp --sport 53
# /sbin/iptables -A INET_OUT_PORT -o eth1 -p tcp --sport 22
# /sbin/service iptables save
# iptables -L INET_IN_PORT -v -n
# iptables -L INET_OUT_PORT -v -nTask: IP Accounting by Protocol
Set ip accounting for tcp, udp, and icmp protocols on eth1:# /sbin/iptables -N INET_IN_PROTOCOL
# /sbin/iptables -N INET_OUT_PROTOCOL
### ATTACH them (USE FORWARD for gateway systems) ###
# /sbin/iptables -A INPUT -j INET_IN_PROTOCOL
# /sbin/iptables -A OUTPUT -j INET_OUT_PROTOCOL
### Match tcp, udp and icmp ###
# /sbin/iptables -A INET_IN_PROTOCOL -i eth1 -m tcp -p tcp
# /sbin/iptables -A INET_IN_PROTOCOL -i eth1 -m udp -p udp
# /sbin/iptables -A INET_IN_PROTOCOL -i eth1 -m icmp -p icmp
# /sbin/iptables -A INET_OUT_PROTOCOL -o eth1 -m tcp -p tcp
# /sbin/iptables -A INET_OUT_PROTOCOL -o eth1 -m udp -p udp
# /sbin/iptables -A INET_OUT_PROTOCOL -o eth1 -m icmp -p icmp
### save them (redhat and friends only) ###
# /sbin/service iptables saveTo view stats, enter:
# iptables -L INET_OUT_PROTOCOL -v -n
# iptables -L INET_IN_PROTOCOL -v -nHow Do I View Exact Value Of the Packet and Byte Counters?
Use the -x option:# iptables -L {Chain-Name-Here} -x
# iptables -L INET_IN_PORT -v -x
# iptables -L INET_IN_PORT -v -x -nSample outouts:
Chain INET_IN_PORT (1 references)
pkts bytes target prot opt in out source destination
274688 45080261 tcp -- eth1 any anywhere anywhere tcp dpt:http
17 1517 tcp -- eth1 any anywhere anywhere tcp dpt:https
0 0 tcp -- eth1 any anywhere anywhere tcp dpt:ssh
0 0 udp -- eth1 any anywhere anywhere udp dpt:snmp
3 228 udp -- eth1 any anywhere anywhere udp dpt:ntp
How Do Delete Packet and Byte Counters?
Use the -Z option:# iptables -L {CHAIN-NAME-HERE} -Z
# iptables -L INET_IN_PORT -ZVerify the same, enter:
# iptables -L INET_IN_PORT -vnxHow Do I View IP Accounting (Bandwidth Traffic) In Real Time?
You can execute iptables periodically, showing output fullscreen using the watch command as follows:# watch -n -v -L INET_OUT -n
# watch -n 1 iptables -v -L INET_INSample outputs:
Fig.03: Portwise IP Stats
Fig.04: Linux Apache IP accounting
Please note that I've removed domain names from the above output.
Fig.05: Outgoing IP Accounting By Protocol
How Do I Save and Restore Counter Across Reboots?
To save counter use the following command:# iptables-save -c > /path/to/iptables.rules
# iptables-restore -c < /path/to/iptables.rulesYou can add iptables-restore command to /etc/rc.local or write a sys v style script (recommended):
#!/bin/sh
#
# save-restore-iptables-counter
#
# chkconfig: 2345 08 92
# description: Save and restore iptables counters
# processname: none
# Note: See how to use this script :
# http://www.cyberciti.biz/faq/linux-configuring-ip-traffic-accounting/
# Source function library.
. /etc/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
path="/etc/sysconfig/iptables.rules.counter"
_restorecmd="/sbin/iptables-restore"
_savecmd="/sbin/iptables-save"
prog="save-restore-iptables-counter"
start() {
echo -n $"Starting $prog: "
$_restorecmd -c < "$path"
retval=$?
return $retval
}
stop() {
echo -n $"Stopping $prog: "
$_savecmd -c > "$path"
retval=$?
return $retval
}
restart(){
stop
start
}
case "$1" in
start)
start;;
stop)
stop;;
restart)
restart;;
*)
echo $"Usage: $0 {start|stop|restart}"
exit 3
esac
Sample Redhat / CentOS / Fedora Specific Config File - /etc/sysconfig/iptables
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [2027:704756]
:INET_IN - [0:0]
:INET_OUT - [0:0]
:INET_IN_PORT - [0:0]
:INET_OUT_PORT - [0:0]
:INET_IN_PROTOCOL - [0:0]
:INET_OUT_PROTOCOL - [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j INET_IN
-A INPUT -j INET_IN_PORT
-A INPUT -j INET_IN_PROTOCOL
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A OUTPUT -j INET_OUT
-A OUTPUT -j INET_OUT_PORT
-A OUTPUT -j INET_OUT_PROTOCOL
# IP accounting apache
-A INET_IN -d 123.1.2.1
-A INET_OUT -s 123.1.2.1
# Port wise ip accouting
-A INET_IN_PORT -i eth0 -p tcp -m tcp --dport 80
-A INET_OUT_PORT -o eth0 -p tcp -m tcp --sport 80
# protcol wise ip accouting
-A INET_IN_PROTOCOL -i eth0 -m tcp -p tcp
-A INET_IN_PROTOCOL -i eth0 -m udp -p udp
-A INET_IN_PROTOCOL -i eth0 -m icmp -p icmp
-A INET_OUT_PROTOCOL -o eth0 -m tcp -p tcp
-A INET_OUT_PROTOCOL -o eth0 -m udp -p udp
-A INET_OUT_PROTOCOL -o eth0 -m icmp -p icmp
# Unlimited to lo
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
# ICMP
-A RH-Firewall-1-INPUT -i eth0 -p icmp --icmp-type echo-reply -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -p icmp --icmp-type destination-unreachable -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -p icmp --icmp-type time-exceeded -j ACCEPT
# ICMP ping reply
-A RH-Firewall-1-INPUT -i eth0 -p icmp --icmp-type echo-request -j ACCEPT
# Drop spoof
-A RH-Firewall-1-INPUT -i eth0 -s 10.0.0.0/8 -j LOG --log-prefix "IP DROP SPOOF A: "
-A RH-Firewall-1-INPUT -i eth0 -s 10.0.0.0/8 -j DROP
-A RH-Firewall-1-INPUT -i eth0 -s 172.16.0.0/12 -j LOG --log-prefix "IP DROP SPOOF B: "
-A RH-Firewall-1-INPUT -i eth0 -s 172.16.0.0/12 -j DROP
-A RH-Firewall-1-INPUT -i eth0 -s 192.168.0.0/16 -j LOG --log-prefix "IP DROP SPOOF C: "
-A RH-Firewall-1-INPUT -i eth0 -s 192.168.0.0/16 -j DROP
-A RH-Firewall-1-INPUT -i eth0 -s 224.0.0.0/4 -j LOG --log-prefix "IP DROP MULTICAST D: "
-A RH-Firewall-1-INPUT -i eth0 -s 224.0.0.0/4 -j DROP
-A RH-Firewall-1-INPUT -i eth0 -s 224.0.0.0/5 -j LOG --log-prefix "IP DROP SPOOF E: "
-A RH-Firewall-1-INPUT -i eth0 -s 224.0.0.0/5 -j DROP
-A RH-Firewall-1-INPUT -i eth0 -s 127.0.0.0/8 -j LOG --log-prefix "IP DROP LOOPBACK "
-A RH-Firewall-1-INPUT -i eth0 -s 127.0.0.0/8 -j DROP
# Drop INVALID packets immediately
-A INPUT -i eth0 -p ALL -m state --state INVALID -j LOG --log-prefix "IP DROP INVALID "
-A INPUT -i eth0 -p ALL -m state --state INVALID -j DROP
-A INPUT -i eth0 -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "IP DROP SYN "
-A INPUT -i eth0 -p tcp ! --syn -m state --state NEW -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags ALL NONE -j LOG --log-prefix "IP DROP TCPFLAGS:1 "
-A INPUT -i eth0 -p tcp --tcp-flags ALL NONE -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags ALL ALL -j LOG --log-prefix "IP DROP TCPFLAGS:2 "
-A INPUT -i eth0 -p tcp --tcp-flags ALL ALL -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags ALL FIN,URG,PSH -j LOG --log-prefix "IP DROP TCPFLAGS:3 "
-A INPUT -i eth0 -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j LOG --log-prefix "IP DROP TCPFLAGS:4 "
-A INPUT -i eth0 -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "IP DROP TCPFLAGS:5 "
-A INPUT -i eth0 -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A INPUT -i eth0 -p tcp --tcp-flags SYN,FIN SYN,FIN -j LOG --log-prefix "IP DROP TCPFLAGS:6 "
-A INPUT -i eth0 -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
-A INPUT -i eth0 --fragment -p ICMP -j LOG --log-prefix "IP DROP ICMP FRAGMENT "
-A INPUT -i eth0 --fragment -p ICMP -j DROP
# Open port and other stuff
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -s 124.1.2.3 -i eth0 -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT
-A RH-Firewall-1-INPUT -s 124.1.2.3 -i eth0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 25 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 143 -j ACCEPT
-A RH-Firewall-1-INPUT -s 0/0 -i eth0 -p tcp -m state --state NEW -m tcp --dport 110 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
#end
Syntax Summary
Use the following syntax for gateway system IP accounting:iptables -N YOUR-CHAIN-HEREUse the following syntax for a dedicated server roles or complex hosting solutions:
iptables -A FORWARD -j YOUR-CHAIN-HERE
iptables -A YOUR-CHAIN-HERE -d IP-ADDRESS
iptables -A YOUR-CHAIN-HERE -s IP-ADDRESS
iptables -A YOUR-CHAIN-HERE -i ppp0 -p tcp -m tcp --sport PORT
iptables -A YOUR-CHAIN-HERE -o ppp0 -p tcp -m tcp --dport PORT
iptables -A YOUR-CHAIN-HERE -i ppp0 config-options-as-per-requirements
iptables -N YOUR-INPUT-CHAIN-HERE
iptables -N YOUR-OUTPUT-CHAIN-HERE
iptables -A INPUT -j YOUR-INPUT-CHAIN-HERE
iptables -A OUTPUT -j YOUR-OUTPUT-CHAIN-HERE
iptables -A YOUR-INPUT-CHAIN-HERE -d IP-ADDRESS
iptables -A YOUR-OUTPUT-CHAIN-HERE -s IP-ADDRESS
iptables -A YOUR-CHAIN-HERE -i eth1 config-options-as-per-requirements
Conclusion
iptables provides an excellent way to track ip accounting including bandwidth usage. You can monitor and track subnets, ports, protocols, bandwidth usage for file sharing, apache, email and much more. You can use tc to control and shape traffic. You can enhance your IP accounting as follows:- Write a shell / perl script to create monthly reports.
- Use RRDTool to generate graphs.
- Collect data from all servers and save it to mysql database using perl or php.
- Use ip6tables to create IPv6 specific IP accounting reports.
Recommend readings:
See the iptables / ip6tables command man page for more information:man iptables
man ip6tables
No comments:
Post a Comment