ARP Cache Issues In OpenStack Environments

Jan 16, 2026 by Editorial Team 43 views

ARP Cache Overflow in OpenStack: A Deep Dive

Hey guys, let's talk about a tricky issue that can pop up in large OpenStack environments: ARP cache overflow. This problem can lead to some seriously annoying network connectivity issues, and it's something we need to understand and address.

The Problem: ARP Cache Saturation

So, what's the deal with the ARP cache? Well, it's a table on your network devices (like servers) that stores the mapping between IP addresses and their corresponding MAC addresses. This table helps your devices quickly figure out where to send network traffic. The problem arises when this table gets too full. On Ubuntu, for example, the default maximum size is set to 1024 entries. When you're running a bunch of containers, like in a large OpenStack setup, this limit can be easily exceeded. The result? You'll start seeing errors like "neighbor: arp_cache: neighbor table overflow!" in your system logs.

Why cos-proxy is a Key Contributor

Now, in this scenario, a charm called cos-proxy seems to be a major player in creating these ARP cache entries. This makes sense since metrics are being collected through cos-proxy, and adding new compute nodes leads to a proportional increase in entries. In a 100+ node environment, the cos-proxy units can quickly become top contenders for the most ARP entries. In this article, we'll look at ways to better manage this situation. We want to decrease the number of ARP entries. We'll dive into what causes this issue, how to identify it, and some possible solutions.

Identifying the Issue

To see if you're hitting this ARP cache limit, you can use a few handy commands. First, SSH into a machine hosting your cos-proxy container. Then, run this command to see the number of ARP entries for each LXD unit:

for i in `sudo lxc list -c n -f csv`; do echo $i ;sudo lxc exec $i -- ip neigh show | wc -l; done

This will give you a list of your LXD units and the number of ARP entries each one has. The output from the report in the original bug report shows us that the cos-proxy charm often has a high number of entries. This output shows that cos-proxy can easily be at the top of the list for ARP entries. Other charms also contribute to the total, but cos-proxy is a significant factor.

To get a sense of the total ARP entries on a single machine, you can run this command and then sum the results:

for i in `sudo lxc list -c n -f csv`; do sudo lxc exec $i -- ip neigh show | wc -l; done > sum.txt
awk '{s+=$1} END {print s}' ./sum.txt

If the final number is close to or exceeds the default limit of 1024, you're likely facing the ARP cache overflow issue. The examples above show that the sum of the entries can easily exceed the default threshold in a real-world scenario. This can lead to the "neighbor table overflow" error and subsequent network connectivity issues.

Relevant Log Output

Look out for the "neighbor: arp_cache: neighbor table overflow!" error in your system logs (usually /var/log/syslog). This is the smoking gun, and it means your ARP cache is full, and new entries are not being added.

The Root Causes

So, what causes this flood of ARP entries? Well, in a large environment with lots of containers, each container communicates with other resources and services. This communication generates ARP requests and responses, creating entries in the ARP cache. The cos-proxy charm plays a vital role in collecting metrics. Each time it interacts with new endpoints, it needs to resolve the MAC addresses for the IP addresses of those endpoints, which populates the ARP cache.

Other Contributing Factors

It's important to remember that cos-proxy isn't the only charm contributing to this. Other charms, like rabbitmq-server, ovn-central, and cinder (as seen in the provided list) also generate ARP entries. Every service and container that interacts with other network resources contributes to the total number of ARP cache entries.

Possible Solutions and Mitigation Strategies

Okay, so we've identified the problem and understand the causes. Now, how do we fix it? Here are a few approaches you can take.

Increase the ARP Cache Size

The easiest (but perhaps not the best) solution is to increase the maximum size of the ARP cache. You can do this by modifying the /proc/sys/net/ipv4/neigh/default/gc_thresh3 value. You can refer to this guide for specific commands. Keep in mind that increasing the cache size just pushes the problem further down the road. You're delaying the inevitable, but you're not actually solving the root issue.

Optimize Network Configuration

Another approach is to optimize your network configuration. This might involve:

Reducing the frequency of ARP requests: Analyze your network traffic and identify any unnecessary ARP requests. Reducing these requests can help alleviate the pressure on your ARP cache. This can be done by adjusting ARP timeouts and other network parameters.
Using static ARP entries: For critical services, consider using static ARP entries to prevent the need for dynamic ARP lookups. Static entries bypass the ARP process altogether, reducing the load on your cache. However, you need to be careful with this approach, as manually managing static entries can be complex and prone to errors.

Investigate the `cos-proxy` Configuration

Since cos-proxy is a major contributor in the generated ARP cache entries, reviewing its configuration can be beneficial:

Metric Collection Frequency: Can the collection frequency be adjusted? Perhaps less frequent metric collection would help reduce the number of ARP requests.
Network Segmentation: Implement network segmentation to limit the scope of ARP broadcasts. This way, each container will only need to resolve MAC addresses within its segment, reducing the overall number of entries.

Regular Monitoring and Alerting

Implement proper monitoring and alerting on your ARP cache. Set up alerts to notify you when the cache size approaches its limit. This will give you time to react before network connectivity issues arise.

The Importance of a Long-Term Solution

While increasing the ARP cache size can provide temporary relief, it doesn't address the underlying issue. The ideal solution would be to reduce the number of ARP entries generated in the first place. You need to investigate the communication patterns within your environment and identify ways to minimize unnecessary ARP requests.

Conclusion

Guys, dealing with ARP cache overflow in large OpenStack environments can be tricky. Identifying the problem, understanding the root causes, and implementing appropriate solutions will help you maintain a stable and reliable network. Monitoring, analyzing, and optimizing your network configuration are key. Remember to regularly monitor your ARP cache, adjust your network parameters, and consider long-term solutions to prevent this issue from resurfacing. With a little effort, you can keep your network humming along smoothly!