Operations

Daily checks

Use systemctl status, journalctl, /var/log/syslog, nft list ruleset and ip route to inspect the local state. Check CPU, memory, disk space and conntrack usage before investigating a network symptom.

Routing checks

Use vtysh -c "show ip route", vtysh -c "show ip ospf neighbor" and vtysh -c "show bgp summary" when applicable. A missing route is usually easier to diagnose from the FRR neighbor state first.

Publishing a service

Add the service host to the appropriate nftables host group, add its port to the matching port group and add the forward rule. Test from an external network and confirm the packet path on the WAN and LAN interfaces.

HA maintenance

Change the standby node first, verify it, then perform a controlled failover before updating the former active node. Never update both firewalls at the same time.

Recovery

A clean Debian host can be rebuilt from the repository, inventory and Vault secrets. Runtime state such as conntrack sessions, OSPF adjacencies and VPN handshakes is recreated after the playbook runs.

Troubleshooting order

Start at the packet path: DNS resolution, route selection, WAN capture, firewall counters, LAN capture, then the service itself. This avoids changing nftables when the real problem is routing, NAT or an unavailable backend.

Firewall inspection

Start with the generated policy and counters. Use nft list ruleset to confirm the rule exists, nft list chain inet filter forward to inspect counters, and conntrack -L to see whether the flow is tracked. A counter at zero means the packet did not reach that rule.

sudo nft list ruleset
sudo conntrack -S
sudo journalctl -u nftables -u frr -u keepalived

FRR diagnosis

Use vtysh to inspect protocol state without exposing VTY Telnet ports. Check the neighbor state, selected route and installed kernel route. If OSPF is down, verify interface, area, network type, MTU and authentication before changing timers.

Failover procedure

Drain or demote the active node, confirm the VIP moves, then test an existing NAT connection and a new connection. After maintenance, restore the preferred priority and verify that Conntrackd, Keepalived and FRR converge without a split-brain state.

Safe change procedure

Record the intended flow or service change, run syntax validation and check mode, then apply to the standby node first. Confirm services, routes, counters and logs before failing over. Keep the previous variables available for rollback and never update both HA nodes simultaneously.

Packet troubleshooting

Follow the packet from client to ingress interface, nftables counters, route lookup, egress interface and backend. Use tcpdump at two points in the path and compare timestamps. If the packet arrives but no counter moves, inspect the interface and chain. If both sides see traffic but the session fails, inspect NAT and return routing.

Recovery expectations

A rebuilt Debian host receives its intended state from inventory, variables and Vault secrets. Recheck interface names, system identity, routing neighbors, VPN handshakes, VIP ownership and conntrack behavior after restoration. Application sessions that are not represented in conntrack must be re-established.