Monitoring Bandwidth Usage: Python's Essential Guide

Bandwidth usage monitoring is essential for individuals and businesses to understand their data consumption and make informed decisions about their internet plans. While it was not a significant concern during the modem era, the rise of high-speed connections and expectations of unlimited bandwidth have made it crucial. Python offers various tools and modules to monitor bandwidth usage, such as psutil, nload, and tcptrack. Additionally, there are stand-alone tools like Networxs, Freemeter, and BitMeter OS that can be used for this purpose. Python scripts can also be utilized to develop custom solutions, providing flexibility and control over the monitoring process.

Using Speedtest.net servers

There are several ways to monitor bandwidth usage in Python, and one popular method is to use the speedtest-cli package, which provides a command-line interface for testing internet bandwidth using speedtest.net servers.

To install the speedtest-cli package, you can use the following command:

Pip install speedtest-cli

Once installed, you can use the speedtest-cli command to perform various tests, such as download and upload speed tests, and obtain detailed information about your internet connection.

For example, to perform a basic speed test, you can use the following command:

Speedtest-cli

This will connect to a nearby speedtest.net server and perform a series of downloads and uploads to measure your connection's speed. The results will be displayed on the terminal, including information such as ping, download speed, upload speed, and the server used for the test.

You can also specify additional options to customise the speed test. For instance, to test against a specific server, you can use the --server option:

Speedtest-cli --server SERVER_ID

Replace SERVER_ID with the ID of the desired server. You can obtain a list of available speedtest.net servers using the --list option:

Speedtest-cli --list

This will display a list of servers sorted by their distance from your location.

Additionally, if you want to perform only a download or upload test, you can use the --no-upload or --no-download options, respectively:

Speedtest-cli --no-upload

Speedtest-cli --no-download

These options allow you to measure the performance of your connection for specific types of data transfer.

The speedtest-cli package provides a flexible and powerful way to monitor and test your internet bandwidth using speedtest.net servers. By utilising this package, you can gain valuable insights into your connection's speed and performance, helping you make informed decisions about your network usage.

Monitor network traffic per executable

Monitoring network traffic per executable involves using Python libraries like psutil and Scapy to retrieve network statistics and established connections. Here's a detailed guide on how to achieve this:

Installation and Imports:

First, ensure you have the required libraries installed. You can install them using pip:

Python

Pip install psutil scapy pandas

Then, import the necessary modules:

Python

Import psutil

Import time

Import os

Import pandas as pd

From scapy.all import *

From collections import defaultdict

From threading import Thread

Understanding the Data:

Before proceeding, it's important to understand the data you'll be working with. Here's a breakdown:

• `psutil.net_io_counters()`: This function returns network input and output statistics. By setting `pernic=True`, you can get these statistics for each network interface.
• `psutil.net_connections()`: This function provides information about each connection's source and destination ports, as well as their process ID (PID).
• `pid2traffic`: A dictionary that maps each process ID (PID) to a list of two values representing the upload and download traffic.
• `connection2pid`: A dictionary that maps each connection (represented by source and destination ports) to its corresponding process ID (PID).
• `all_macs`: A set containing the MAC addresses of all network interfaces on your machine.

Packet Sniffing with Scapy:

Scapy is a powerful packet manipulation library. To monitor network traffic, you'll use its `sniff()` function, which captures packets from the network. Here's how you can set up packet sniffing:

Python

Def process_packet(packet):

Global pid2traffic

Try: packet_connection = (packet.sport, packet.dport)

Except (AttributeError, IndexError): pass

Else: packet_pid = connection2pid.get(packet_connection)

If packet_pid: if packet.src in all_macs: pid2traffic[packet_pid][0] += len(packet) # Outgoing packet (upload) else: pid2traffic[packet_pid][1] += len(packet) # Incoming packet (download)

Sniff(prn=process_packet, store=False)

Getting Connections:

To monitor network traffic per executable, you need to keep track of the connections on your machine. Here's a function to do that:

Python

Def get_connections(): global connection2pid while is_program_running: for c in psutil.net_connections(): if c.laddr and c.raddr and c.pid: connection2pid[(c.laddr.port, c.raddr.port)] = c.pid connection2pid[(c.raddr.port, c.laddr.port)] = c.pid time.sleep(1)

Calculating and Printing Network Usage:

Now, you can calculate and print the network usage per process:

Python

Def print_pid2traffic(): global global_df processes = [] for pid, traffic in pid2traffic.items(): try: p = psutil.Process(pid) name = p.name() create_time = datetime.fromtimestamp(p.create_time()) except psutil.NoSuchProcess: continue process = { "pid": pid, "name": name, "create_time": create_time, "Upload": traffic [0], "Download": traffic [1], } try: process ["Upload Speed"] = traffic [0] - global_df.at [pid, "Upload"] process ["Download Speed"] = traffic [1] - global_df.at [pid, "Download"] except (KeyError, AttributeError): process ["Upload Speed"] = traffic [0] process ["Download Speed"] = traffic [1] processes.append(process) df = pd.DataFrame(processes) df = df.set_index("pid") df.sort_values("Download", inplace=True, ascending=False) printing_df = df.copy() printing_df ["Download"] = printing_df ["Download"].apply(get_size) printing_df ["Upload"] = printing_df ["Upload"].apply(get_size) printing_df ["Download Speed"] = printing_df ["Download Speed"].apply(get_size).apply(lambda s: f"{s}/s") printing_df ["Upload Speed"] = printing_df ["Upload Speed"].apply(get_size).apply(lambda s: f"{s}/s") os.system("cls") if "nt" in os.name else os.system("clear") print(printing_df.to_string()) global_df = df

Def print_stats(): while is_program_running: time.sleep(1) print_pid2traffic()

Main Function:

Finally, in the main function, you start the threads for printing statistics and getting connections:

Python

If __name__ == "__main__": printing_thread = Thread(target=print_stats) printing_thread.start() connections_thread = Thread(target=get_connections) connections_thread.start()

Running the Sniffer:

To start capturing packets and monitoring network traffic, you can use the following code:

Python

Print("Started sniffing") sniff(prn=process_packet, store=False) is_program_running = False

This guide provides a comprehensive overview of monitoring network traffic per executable using Python. You can adapt and extend the code to suit your specific needs and requirements.

Using psutil

The Python system utilities module, `psutil`, can be used to monitor bandwidth usage. It is a cross-platform library that retrieves information on running processes, system and hardware information.

To install `psutil`, type the following into the terminal:

$ pip3 install psutil

To get started, import `psutil` and define the network interface:

Python

Import psutil

NETWORK_INTERFACE = 'ppp0'

Then, you can use the net_io_counters function to get the total number of bytes sent and received:

Python

Netio = psutil.net_io_counters(pernic=True)

Net_usage = netio [NETWORK_INTERFACE].bytes_sent + netio [NETWORK_INTERFACE].bytes_recv

If the output is zero, you may need to change the `NETWORK_INTERFACE` to `'wwan0'.

To calculate the speed, you can subtract the old network stats from the new stats:

Python

While True:

Sleep for 1 second

Time.sleep(1)

Get the network I/O stats again

Io_2 = psutil.net_io_counters()

New - old stats gets us the speed

Us, ds = io_2.bytes_sent - bytes_sent, io_2.bytes_recv - bytes_recv

You can also use `psutil` to monitor bandwidth usage per network interface. Here is an example script that does this:

Python

Import psutil

Import time

Import os

Import pandas as pd

UPDATE_DELAY = 1 # in seconds

Def get_size(bytes):

"""Returns size of bytes in a nice format"""

For unit in ['', 'K', 'M', 'G', 'T', 'P']:

If bytes < 1024:

Return f"{bytes:.2f}{unit}B"

Bytes /= 1024

While True:

Sleep for `UPDATE_DELAY` seconds

Time.sleep(UPDATE_DELAY)

Get the network I/O stats from psutil on each network interface

By setting `pernic` to `True`

Io = psutil.net_io_counters(pernic=True)

Initialize the data to gather (a list of dicts)

Data = []

For iface, iface_io in io.items():

New - old stats gets us the speed

Upload_speed, download_speed = iface_io.bytes_sent - io.bytes_sent, iface_io.bytes_recv - io.bytes_recv

Data.append({

"iface": iface,

"Download": get_size(iface_io.bytes_recv),

"Upload": get_size(iface_io.bytes_sent),

"Upload Speed": f"{get_size(upload_speed / UPDATE_DELAY)}/s",

"Download Speed": f"{get_size(download_speed / UPDATE_DELAY)}/s",

})

Update the I/O stats for the next iteration

Io = io_2

Construct a Pandas DataFrame to print stats in a tabular style

Df = pd.DataFrame(data)

Sort values per column

Df.sort_values("Download", inplace=True, ascending=False)

Clear the screen based on your OS

Os.system("cls") if "nt" in os.name else os.system("clear")

Print the stats

Print(df.to_string())

Using nload or tcptrack

There are a few options for monitoring bandwidth usage in Python, one of which is to use the nload or tcptrack tools. These are command-line tools that can display network usage information in real time.

To use nload, you would first need to install it on your system. Once installed, you can run the nload command in your terminal to see a real-time display of network usage. By default, nload will show the usage for all network interfaces on your system. You can also specify a particular interface to monitor by using the -i option, followed by the name of the interface. For example, to monitor the eth0 interface, you would use the command nload -i eth0.

Similarly, tcptrack is a command-line tool that can be used to monitor TCP connections on a network. It displays information such as the state of connections, source/destination addresses, and bandwidth usage. To use tcptrack, you need to specify the network interface you want to monitor using the -i option. For example, to monitor the eth0 interface, you would use the command tcptrack -i eth0. You can also filter the connections displayed by using a pcap filter expression as an argument. For example, to only show connections from a specific host, you can use the command tcptrack -i eth0 src or dst [host address].

In addition to these command-line tools, there are also Python libraries available that can be used to monitor bandwidth usage. For example, the psutil library can be used to get the total number of bytes sent and received over the network. You can then use this information to calculate the network speed or bandwidth usage. Another option is to use the iftop library, which is a command-line tool for displaying network usage information in real time.

Using iptables and MySQL

To monitor bandwidth usage in Python, you can use iptables and MySQL. Here's a step-by-step guide:

Step 1: Set Up Iptables Rules

Create iptables rules to monitor network traffic. You can use the following commands:

/usr/sbin/iptables -nvx -L INPUT_LAN

/usr/sbin/iptables -nvx -L OUTPUT_LAN

/usr/sbin/iptables -nvx -L FORWARD_IN_LAN

/usr/sbin/iptables -nvx -L FORWARD_OUT_LAN

These commands will display the input, output, and forwarded traffic on your network, allowing you to monitor bandwidth usage.

Step 2: Parse Data with Bash

Use Bash to parse the output of the iptables commands. You can redirect the output to log files for further processing:

$IPTABLES -nvx -L INPUT_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,9,12 > $LOG_IN

$IPTABLES -nvx -L OUTPUT_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,10 > $LOG_OUT

$IPTABLES -nvx -L FORWARD_IN_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,9,12 > $LOG_F_IN

$IPTABLES -nvx -L FORWARD_OUT_LAN | tr -s [:blank:] | grep "192." | cut -d' ' -f3,10 > $LOG_F_OUT

Step 3: Store Data in MySQL Database

Create a Python script to read the log files generated in Step 2 and insert the data into a MySQL database. Here's an example script:

Python

#!/usr/bin/python

Import MySQLdb as sql

Import sys

If len(sys.argv) < 4:

Raise AssertionError("4 Filenames Required")

Conn = sql.connect('localhost', 'username', 'password', db='userdb')

Cursor = conn.cursor()

InetData = {}

InputFile = open(sys.argv[1], 'r')

OutputFile = open(sys.argv[2], 'r')

ForwardInFile = open(sys.argv[3], 'r')

ForwardOutFile = open(sys.argv[4], 'r')

For val in inputFile.readlines():

Val = val.split()

InetData[val[1]] = [int(val[0]), 0, val[2]]

For val in forwardInFile.readlines():

Val = val.split()

InetData[val[1]][0] = inetData[val[1]][0] + int(val[0])

For val in outputFile.readlines():

Val = val.split()

InetData[val[1]][1] = int(val[0])

For val in forwardOutFile.readlines():

Val = val.split()

InetData[val[1]][1] = inetData[val[1]][1] + int(val[0])

For ip, data in inetData.iteritems():

Query = """UPDATE users SET DataRecieved=DataRecieved+%d,DataSent=DataSent+%d WHERE IP="%s" """ % (data [1], data[0], ip)

Cursor.execute(query)

Conn.commit()

Cursor.close()

Conn.close()

InputFile.close()

OutputFile.close()

ForwardInFile.close()

ForwardOutFile.close()

This script reads the log files, processes the data, and updates a MySQL database with the bandwidth usage for each IP address.

Step 4: Regularly Run log.py

To monitor bandwidth usage over time, regularly run the log.py script provided in the GitHub repository. This script will log a database entry each time it is run for each MAC address that was seen during the window. A suitable interval for running the script is every 15 minutes.

By following these steps, you can effectively monitor bandwidth usage using iptables and MySQL in Python.

Frequently asked questions