In this blog post, we will be looking at the NuttX configuration needed for bridged networking from the WIFI interface to the Ethernet interface, using an ESP32 based WT32-ETH01. We'll also be looking at the routing options, and the possibility to route via tunnels.
Note that this is follow-up from the previous post about enabling a dual ETH+WIFI networking stack on the WT32-ETH01.
Bridge as a NuttX application
So, let's start with the bridge example provided in the NuttX apps. It contains two implementation files, one for the hosts, and one for the bridge. The intention is to drive the traffic in this way:
The bridge works by creating two tasks ( using task_create ). Each task if getting it's IP address by invoking the DHCP client (so, the bridge could work with a single network interface). It then listens on an UDP socket, on a given port, and for each packet received, it will forward it to the outgoing IP address. Two tasks are needed as each task is handling unidirectional traffic.
Unfortunately, the bridge example is not of any help. For a bridge to work, we should be able to intercept any kind of traffic, at IP level, and forward it at IP level. The standard way to do that is to use raw sockets, but let's first have a look at uIP networking stack, in case it would already have such bridge capability.
Bridge as a NuttX System Service
Looking at the NuttX net folder, one can notice the ipforward service, described as the "L2 forwarding service".
L2 forwarding
That's exactly what we need - so, let's enable it with this configuration:
Networking Support --->
Internet Protocol Selection --->
[*] Enable L2 forwardingBy default, the services has a table 4 forwarding entries, which get populated dynamically as incoming packet come in. When the stack receives a packet with a destination IP address which does not match the interface IP address, it willl call the ipv4_forward function, described as:
This function is called from ipv4_input when a packet is received that is not destined for us. In this case, the packet may need to be forwarded to another device (or sent back out the same device) depending configuration, routing table information, and the IPv4 networks served by various network devices.
The ipv4_forward function will first try to find the forwarding interface, using netdev_findby_ripv4addr
int ipv4_forward(FAR struct net_driver_s *dev, FAR struct ipv4_hdr_s *ipv4)
{
in_addr_t destipaddr;
in_addr_t srcipaddr;
FAR struct net_driver_s *fwddev;
int ret;
/* Search for a device that can forward this packet. */
destipaddr = net_ip4addr_conv32(ipv4->destipaddr);
srcipaddr = net_ip4addr_conv32(ipv4->srcipaddr);
fwddev = netdev_findby_ripv4addr(srcipaddr, destipaddr);
if (fwddev == NULL)
{
nwarn("WARNING: Not routable\n");
return (ssize_t)-ENETUNREACH;
}Routing Table
From the netdev_findby_ripv4addr one can notice that there is a CONFIG_NET_ROUTE configuration used for the routing table. It is not enabled by default, so let's activate it under:
Networking Support --->
Routing Table Configuration --->
[*] Routing table supportIt also comes with a default of 4 routes, but note that routes are handled at network prefix level, while forwarding entries are at IP level. So, if a route handles a /16 prefix, you could have up to 64K IP to forward.
Unfortunately, the compilation does not work the first time, complaining about missing RTF_XXX definitions.
src/network.c: In function 'wapi_act_route_gw':
src/network.c:141:17: error: 'RTF_UP' undeclared (first use in this function); did you mean 'IFF_UP'?
rt.rt_flags = RTF_UP | RTF_GATEWAY;
^~~~~~
IFF_UP
src/network.c:141:17: note: each undeclared identifier is reported only once for each function it appears in
src/network.c:141:26: error: 'RTF_GATEWAY' undeclared (first use in this function)
rt.rt_flags = RTF_UP | RTF_GATEWAY;
^~~~~~~~~~~
src/network.c:144:22: error: 'RTF_HOST' undeclared (first use in this function)
rt.rt_flags |= RTF_HOST;The issue is actually from the WAPI wireless configuration shell that was enabled for the WIFI configuration. The good news is that it tells us that WAPI can be used to inspect and configure the routes. The bad news is that it seems we will have to manually patch the code for the compilation to work.
The reason for the failure is that WAPI wants to ioctl SIOCADDRT and SIOCDELRT to configure the routing entries. So, let's have a look at how the net routing component handles those two ioctl. That's done in the netdev_ioctl.c file:
#ifdef CONFIG_NET_ROUTE
static int netdev_rt_ioctl(FAR struct socket *psock, int cmd, FAR struct rtentry *rtentry)
{
switch (cmd)
{
case SIOCADDRT: /* Add an entry to the routing table */
return ioctl_add_ipv4route(rtentry);
break;
...
case SIOCDELRT: /* Add an entry to the routing table */
return ioctl_del_ipv4route(rtentry);
break;
}
#endifGood news, both ioctl_del_ipv4route and ioctl_add_ipv4route ignore the rt_flag which WAPI was trying to set, so we can safely comment out the line rt.rt_flags = RTF_UP | RTF_GATEWAY; in WAPI network.c. Those flags are used to indicate if the routing entry is a host or a gateway - and we'll figure out later if this matters to us.
Run-time route configuration
Now that the compilation is working, let's try to configure the routes via WAPI. Here are the logs which can be seen from the NuttShell:
NuttShell (NSH) NuttX-10.0.1
nsh> ipv4_forward: WARNING: Packet forwarding to same device not supported (srcIP=192.168.1.203 dstIP=224.0.0.251)
udp_input: WARNING: No listener on UDP port 5353The message packet forwarding to same device not supported tells us that a packet has been received from "192.168.1.203" and aimed for "224.0.0.251", which is a multicast IP for the mDNS service running on port 5353. This protocol is not enabled by default, hence the "no listener" warning log.
So, let's try to check the routes now with a route command:
nsh> route
SEQ TARGET NETMASK ROUTER
1. 0.0.0.0 0.0.0.0 192.168.1.1The netmask 0 corresponds to a /32 CIDR, meaning that all IP addresses will be considered for this route. As for the router 192.168.1.1, this is the openwrt router, and this is exactly the expected IP. So, first, let's try to add a route
nsh> addroute 8.8.8.8/32 192.168.1.1
nsh> route
SEQ TARGET NETMASK ROUTER
1. 0.0.0.0 0.0.0.0 192.168.1.1
2. 8.8.8.8 255.255.255.255 192.168.1.1Then let's try to ping the newly added destination (8.8.8.8). It works fine because the gateway is properly defined.
nsh> ping 8.8.8.8
PING 8.8.8.8 56 bytes of data
56 bytes from 8.8.8.8: icmp_seq=1 time=190 msNow, let's try to add a route to a non existing gateway. Ping should fail with a "no route available" - not because it can not find the route, but because it can not find a way to reach the gateway (to be precise, because the gateway in on the same subnet as the ESP, the ESP will try to ARP the gateway, and since no one replies, it will fail to send the ICMP packet).
nsh> addroute 4.4.4.4/32 192.168.1.2
nsh> route
SEQ TARGET NETMASK ROUTER
1. 0.0.0.0 0.0.0.0 192.168.1.1
2. 8.8.8.8 255.255.255.255 192.168.1.1
3. 4.4.4.4 255.255.255.255 192.168.1.2
nsh> ping 4.4.4.4
PING 4.4.4.4 56 bytes of data
arp_send_eventhandler: flags: 2000 sent: 0
arp_send: ERROR: arp_wait failed: -116
...
icmp_sendto: ERROR: Not reachable
Bridge Configuration
Now that routes are working, we just need to slight change the ESP configuration, from a Wifi Station to a WIFI Access point, which the Mac will connect to, as described in the diagram below:
There are a few configurations which need to be changed, and the first is to add a DHCP server so that the MAC can get an IP address when connecting to the WT32 Access Point (AP):
Application Configuration --->
Network Utilities --->
[*] DHCP serverThe second is to turn the WIFI from a Station to an AP. Let's try to do that with WAPI:
nsh> wapi mode wlan0 WAPI_MODE_MASTERUnfortunately, that does not work, because... well, the underling SIOCSIWMODE is not implemented in the current esp32 driver. Never mind, let's try the reverse bridge, where the Mac is connected via Ethernet:
For this configuration, there should only be a need for the DHCP server. So, after flashing the new firmware configuration, and connecting the Mac to the ESP via the ethernet cable, I could see the following logs:
udp_input: WARNING: No listener on UDP port 67That just tells us that the DHCP Server is not started. Looking at the code, the missing link seems that we also need to enable the server from the app config.
Application Configuration --->
Examples --->
[*] DHCP server exampleVoila, let's reflash and start the app from the Nutt Shell:
nsh> dhcpd_start eth0
nsh> ifconfig
eth0 Link encap:Ethernet HWaddr a8:03:2a:a1:2c:c5 at UP
inet addr:10.0.0.1 DRaddr:10.0.0.1 Mask:255.255.255.0
wlan0 Link encap:Ethernet HWaddr a8:03:2a:a1:2c:c4 at UP
inet addr:192.168.1.182 DRaddr:192.168.1.1 Mask:255.255.255.0Bingo, it works - the eth0 interface is having its own subnet. A quick check on the Mac confirms that all works fine :-)
⋊> ~/P/m/n/nuttx on master ⨯ ifconfig en5 flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=4<VLAN_MTU>
inet6 fe80::8ca:dbd1:a602:81d6%en5 prefixlen 64 secured scopeid 0xf
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (100baseTX <full-duplex>)
status: activeLast, step, let's try to ping from the Mac and see if it gets through the ESP - for that to work, we should first ensure that the route is properly set on the ESP:
nsh> addroute default 192.168.1.1 wlan0
nsh> route
SEQ TARGET NETMASK ROUTER
1. 0.0.0.0 0.0.0.0 192.168.1.1The first attempt to ping 8.8.8.8 on the mac resulted in a Request timeout. So dumping the the traffic on the OpenWRT router showed and frames were correctly forwarded:
root@OpenWrt:~# tcpdump ether host A8:03:2A:A1:2C:C4 -nS
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
04:03:59.040947 IP 10.0.0.2 > 8.8.8.8: ICMP echo request, id 19945, seq 9, length 64
04:04:00.040736 IP 10.0.0.2 > 8.8.8.8: ICMP echo request, id 19945, seq 10, length 64Only problem, no response... At first glance, it looks like a routing issue on the OpenWRT, which needs to add a route for the 10.x.x.x subnet to the ESP gateway
root@OpenWrt:~# route add -net 10.0.0.0/24 gateway 192.168.1.182After that, it finally works:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=108 time=293.731 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=108 time=352.465 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=108 time=334.467 msWhile, technically, we could say the routing is working, actually, this last step is not a desired one. What should happen is that the ESP should NAT the IP address from the Mac; that's a standard process for routers.
I am not sure of the reason why NuttX does not NAT the outgoing packet - there does not seems to be any trace of "NAT" in the net source code, and ip forward service does not seems to be handling the NAT, but only decrementing the TTL in the IP header. Let's keep this parked fro now, and we'll come back to this NAT configuration issue in a later blog post.
Tunnelling Traffic
Now that routes are working, the next logical step is to tunnel part of the traffic. The standard way in Posix is to create TAP/TUN virtual interfaces, and fortunately for us, NuttX supports such interfaces. But let's keep this investigation for the next blog.
Conclusion
Obviously, the more we dig into NuttX and the more complex it becomes. But at the same time, since NuttX is a Posix compatible system, we can refer to all the online information about Posix to troubleshoot our configuration. That makes it not only much easier, but also much more interesting as all our configuration learnings can be applied to standard Linux too.
Next step, we will be checking the Ethernet driver performance, to check if we can really reach 100Mb/s, as well as the IP forwarding performance. We'll also be looking at setting up the TAP/TUN interface for the tunnelling.
