SEO

Hello, all,

I'm having issues with pf rule matching on an OpenBSD/Sparc system when
I bridge two interfaces, wherein one interface has an IP address
assigned so that clients on the bridged interfaces can access the
OpenBSD machine itself (for DNS).

A helpful person on the PF list said he has a similar setup, but does
not experience the problem I'm having. So I'm starting to suspect it
might be an SBUS/Sparc-specific problem. I'm working with OpenBSD/Sparc
on an SS20, and, if it makes any difference at all, my interfaces are
lebuffer and ledma.

Issue: bridging causes pf to mis-apply frames to the wrong interface.
This is only a problem if I want to filter directionally -- that is,
allow clients on one side of the bridged subnet more access than clients
on the other.

Using tcpdump on le0 and le2 (the two interfaces added to bridge0) shows
traffic arriving and departing on the correct interfaces all of the
time, regardless of bridge state.

However, traffic appearing in pflog (or on pflog0) as matching rules
from the "wrong" interface appears to be due to the bridge:

$ sudo brconfig bridge0 down
$ sudo tcpdump -netttvvv -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0
Feb 18 09:51:51.949840 rule 2/0(match): pass in on le0: 192.168.1.9 >
192.168.1.1: icmp: echo request (id:19bc seq:0) (ttl 64, id 30421)
Feb 18 09:51:51.950030 rule 4/0(match): pass out on le0: 192.168.1.1 >
192.168.1.9: icmp: echo reply (id:19bc seq:0) (ttl 255, id 25154)

This is all correct. Unfortunately, if I:

$ sudo brconfig bridge0 up
$ sudo tcpdump -netttvvv -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0
Feb 18 09:52:29.459668 rule 3/0(match): pass in on le2: 192.168.1.9 >
192.168.1.1: icmp: echo request (id:19bd seq:0) (ttl 64, id 30487)
Feb 18 09:52:29.459838 rule 4/0(match): pass out on le0: 192.168.1.1 >
192.168.1.9: icmp: echo reply (id:19bd seq:0) (ttl 255, id 21188)

Where:
@2 pass in log-all quick on le0 all
@3 pass in log-all quick on le2 all
@4 pass out log-all quick on le0 all

Incidentally, this only happens to traffic TO and FROM the router. The
packets that TRAVERSE the router from one host on le0 to another on le2,
and vice versa, always appear on the correct interface.

I've tried to use bridge rules, but have had even more problems with
those, as applying the following to my bridgename.bridge0:

rule pass in on le0 tag t_lan
rule pass in on le2 tag t_wap

Results in frames that match NEITHER pf rules for 'tagged t_lan' nor
'tagged t_wap.'

So, is this a bug in bridge? For Sparc? For these specific SBUS cards?

More to the point, whether it is or is not a bug, is this behavior
determinate -- that is, with the bridge up:

inbound traffic from le0 to le2, and vice versa, always matches rules
for the correct interfaces
inbound traffic from le0 AND le2 to the router always appears to match
rules for le2
outbound traffic from the router to le0 AND le2 always appears to match
rules for le0

Why? Can I depend on this to always be the case? How do I know which
interface it will pick for the 'outbound,' and which it will pick for
the 'inbound'? Is that a function of which interface has the IP
assigned? Is it a function of the order in which they were added to the
bridge? Is it a function of boot-time discovery order? Interface
number? Etc.?


Thanks in advance,
JMF

JMF wrote:
> Hello, all,
>
> I'm having issues with pf rule matching on an OpenBSD/Sparc system when
> I bridge two interfaces, wherein one interface has an IP address
> assigned so that clients on the bridged interfaces can access the
> OpenBSD machine itself (for DNS).
>
> A helpful person on the PF list said he has a similar setup, but does
> not experience the problem I'm having. So I'm starting to suspect it
> might be an SBUS/Sparc-specific problem. I'm working with OpenBSD/Sparc
> on an SS20, and, if it makes any difference at all, my interfaces are
> lebuffer and ledma.
>
> Issue: bridging causes pf to mis-apply frames to the wrong interface.
> This is only a problem if I want to filter directionally -- that is,
> allow clients on one side of the bridged subnet more access than clients
> on the other.
>
> Using tcpdump on le0 and le2 (the two interfaces added to bridge0) shows
> traffic arriving and departing on the correct interfaces all of the
> time, regardless of bridge state.
>
> However, traffic appearing in pflog (or on pflog0) as matching rules
> from the "wrong" interface appears to be due to the bridge:
>
> $ sudo brconfig bridge0 down
> $ sudo tcpdump -netttvvv -i pflog0
> tcpdump: WARNING: pflog0: no IPv4 address assigned
> tcpdump: listening on pflog0
> Feb 18 09:51:51.949840 rule 2/0(match): pass in on le0: 192.168.1.9 >
> 192.168.1.1: icmp: echo request (id:19bc seq:0) (ttl 64, id 30421)
> Feb 18 09:51:51.950030 rule 4/0(match): pass out on le0: 192.168.1.1 >
> 192.168.1.9: icmp: echo reply (id:19bc seq:0) (ttl 255, id 25154)
>
> This is all correct. Unfortunately, if I:
>
> $ sudo brconfig bridge0 up
> $ sudo tcpdump -netttvvv -i pflog0
> tcpdump: WARNING: pflog0: no IPv4 address assigned
> tcpdump: listening on pflog0
> Feb 18 09:52:29.459668 rule 3/0(match): pass in on le2: 192.168.1.9 >
> 192.168.1.1: icmp: echo request (id:19bd seq:0) (ttl 64, id 30487)
> Feb 18 09:52:29.459838 rule 4/0(match): pass out on le0: 192.168.1.1 >
> 192.168.1.9: icmp: echo reply (id:19bd seq:0) (ttl 255, id 21188)
>
> Where:
> @2 pass in log-all quick on le0 all
> @3 pass in log-all quick on le2 all
> @4 pass out log-all quick on le0 all
>
> Incidentally, this only happens to traffic TO and FROM the router. The
> packets that TRAVERSE the router from one host on le0 to another on le2,
> and vice versa, always appear on the correct interface.
>
> I've tried to use bridge rules, but have had even more problems with
> those, as applying the following to my bridgename.bridge0:
>
> rule pass in on le0 tag t_lan
> rule pass in on le2 tag t_wap

I'm not sure what you're doing there... I didn't put any rules in my
bridgename.bridge0, just "add le0 add le2 up". I put all the firewall
rules in pf.conf. I didn't use any tagging.

> Results in frames that match NEITHER pf rules for 'tagged t_lan' nor
> 'tagged t_wap.'
>
> So, is this a bug in bridge? For Sparc? For these specific SBUS cards?
>
> More to the point, whether it is or is not a bug, is this behavior
> determinate -- that is, with the bridge up:
>
> inbound traffic from le0 to le2, and vice versa, always matches rules
> for the correct interfaces
> inbound traffic from le0 AND le2 to the router always appears to match
> rules for le2
> outbound traffic from the router to le0 AND le2 always appears to match
> rules for le0
>
> Why? Can I depend on this to always be the case? How do I know which
> interface it will pick for the 'outbound,' and which it will pick for
> the 'inbound'?

You can pick this yourself. It doesn't matter which way you do it. You
MUST specify filter rules in both directions for both interfaces in your
pf.conf file! On one interface you'll want to pass in and out
everything. And you do the actual firewall rules on the other
interface. It doesn't matter which interface you do the filtering on,
it's symmetric.

The situation with bridge pf rules is kind of confusing. Let me illustrate:

----------------------
| |
out<-le0->in BRIDGE in<-le2->out
| |
----------------------

So you could have basically
pass in on le0
pass out on le0

then whatever rules you want on le1
pass out on le2
block in on le2
pass in proto tcp from any to any port 25 on le2
etc. etc.

or you could do the opposite for the same effect:
pass in on le0
block out on le0
pass out proto tcp from any to any port 25 on le0
pass in on le2
pass out on le2

I hope this is helpful, maybe you knew this all already and this isn't
the problem. Can you post your entire bridgename.bridge0 and pf.conf
files, as well as explain what exactly the bridge's function is? If
it's just a bridge between two networks and you don't want to filter
then you shouldn't need any rules or tagging at all, just pass in and
out everything. By default it should learn what's where and be
well-behaved, assuming there's no cycles in your network.

-Nathan

Nathan Kennedy wrote:

>> I've tried to use bridge rules, but have had even more problems with
>> those, as applying the following to my bridgename.bridge0:

>> rule pass in on le0 tag t_lan
>> rule pass in on le2 tag t_wap

> I'm not sure what you're doing there... I didn't put any rules in
> my bridgename.bridge0, just "add le0 add le2 up". I put all the
> firewall rules in pf.conf. I didn't use any tagging.

Sorry, I could have been clearer. I explain in more detail later,
but I want different rules for clients on one interface of the
bridge than for clients on the other. Since PF alone seems to get
confused as to which physical interface traffic goes in and out on
for the bridge, I was hoping to have the bridge tell it.


>> inbound traffic from le0 to le2, and vice versa, always matches rules
>> for the correct interfaces inbound traffic from le0 AND le2 to the
>> router always appears to match rules for le2 outbound traffic from
>> the router to le0 AND le2 always appears to match rules for le0

>> Why? Can I depend on this to always be the case? How do I know
>> which interface it will pick for the 'outbound,' and which it
>> will pick for the 'inbound'?

> You can pick this yourself.

How? Right now, it picks the "correct" interface for traffic
traversing the bridge, and picks le2 as the inbound interface for
all traffic NOT crossing the bridge, and le0 as the outbound for all
traffic NOT originating from the bridge. What did I do to "pick"
this configuration?


> It doesn't matter which way you do it. You MUST specify filter
> rules in both directions for both interfaces in your pf.conf
> file!

If you see my logs later, I'm not sure why I "must." Stateless
traffic appears to require rules for both directions, both
interfaces, but stateful traffic only seems to log against two
rules -- those that established the connection.

Of course, two rules only allows connections to be opened from one
specific side of the bridge to the other, but then, that's one of
the things I'm after. :-)

> On one interface you'll want to pass in and out everything. And
> you do the actual firewall rules on the other interface. It
> doesn't matter which interface you do the filtering on, it's
> symmetric.

> ----------------------
> | |
> out<-le0->in BRIDGE in<-le2->out
> | |
> ----------------------

Yes, I get that. Mine is a little more complex, in that I'm asking
two additional functions out of the device. Most of the literature,
it seems, focuses on the notion of a box that acts as a
"transparent" filtering bridge.

For my application, I want the two networks on either side of the
bridge to have this transparency when communicating between each
other, but I also have the OpenBSD box acting as NAT on a third
interface, DNS to the two bridged interfaces (so it must have an
IP), and to be accessible for certain other services (e.g. SSH) to
only ONE of the bridged interfaces. Thus, I DON'T want to just pass
everything on, say, le2, and filter only on le0, because there is
traffic that is not only crossing the bridge, but going TO the bridge.

-------------------------
| |
| /->}BRIDGE}<-\ |
| | | |
| v v |
out<-le0-->in--->DNS<---in<--le2->out
| ^ | ^ |
| | +->SSH | |
| | | | |
| | \->HTTP | |
| | v |
| \-----FWD<--------NAT<-le1->out
| |
-------------------------

Noting that the box will filter traffic differently on le0 and
le2. (e.g. le2 should get DNS from the box, but will not be able to
SSH into it, or into machines on le0, but the box and machines on
le0 will be able to SSH into machines on le2.)

As I said, if I use stateful rules, it appears this can be done
fairly easily. My problem isn't that the rules aren't can't do what
I want, in theory, it's that the pf/bridge combination seems unable
to determine the physical interfaces for traffic to and from the
OpenBSD machine itself.

When traffic bound for the OpenBSD box hits pf, it appears as though
it comes from le2. That is, when I ping OpenBSD from a machine on
le0, it matches rules for "in on le2."


> Can you post your entire bridgename.bridge0 and pf.conf files, as
> well as explain what exactly the bridge's function is?

Certainly. I do so in just a paragraph or so... actually, I've
stripped out comments, and rules relating to blocking on my third
(NAT) interface, as they don't come into play.


> If it's just a bridge between two networks and you don't want to
> filter then you shouldn't need any rules or tagging at all, just
> pass in and out everything. By default it should learn what's
> where and be well-behaved, assuming there's no cycles in your
> network.

I do want to filter, I want to filter asymmetrically, and it IS
MOSTLY well-behaved. As for "cycles" in the network, I haven't
any. There's a switch on le0, a hub on le2, and only simple clients
with a single interface are plugged into either.

As to what I'm trying to accomplish, this is for adding a wireless
access point to a basic home setup with a cable/DSL connection. I
don't want to hang the AP in a DMZ, because (a) I want to provide
the same level of "protection" to the AP as I do to the LAN, and (b)
because there are certain services (e.g. iTunes autodiscovery) that
I want to be functional between the LAN and AP.


# OpenBSD box with 3 interfaces
#
# le0 is the internal wired LAN
# le1 is the external internet
# le2 is the internal wireless AP
#
# Goals:
#
# Perform NAT for both internal networks to the outside world
# Bridge LAN and wireless AP to provide certain "broadcast-based"
# services (Rendezvous/iTunes, etc.) the impression that they
# span the network
# Provide DNS service to LAN and wireless
# Provide "full" internet functionality to LAN and wireless
# Provice "full" access to machines on the wireless from the LAN
# Provide "limited" services on LAN machines to wireless clients
# "Protect" the router, LAN, and AP from the internet
# "Protect" the router and LAN from the wireless

As you can see, I want the two internal interfaces to be asymmetrical
with respect to what they can do to each other, and with what
services they can access on the OpenBSD box. Additionally, simply
using two separate subnets won't work, because some services use
broadcast messages to discover other nodes. For example, iTunes
broadcasts UDP packets on 224.x.x.x.

Of course, I started with my "best guess" ruleset, and quickly
learned that I had no idea what I was really doing. :-)

So, I decided to use some simple rulesets, with logging, to get a
better understanding of how this is all working. Start by passing
and logging everything, and write rules accordingly to pick off the
traffic that I *do* want to pass, then drop the pass-all rules in
favor of block-all. Sounds reasonable, right?

I'll spare you the miserable failure of my first attempt, and jump
right into the testing setups:

First, I tried:

/etc/hostname.le0
inet 192.168.1.1 255.255.255.0 NONE

/etc/hostname.le1
dhcp NONE NONE NONE

/etc/hostname.le2
up

/etc/bridgename/bridge0
add le0
add le2
up

with PF rules (I've not listed my tables or altq rules for the
external interface, as they do not come into play here):

set optimization normal
set block-policy return

scrub in all fragment reassemble
scrub out on $ext random-id

nat on $ext from 192.168.1.0/24 to any -> $ext

rdr pass on $ext inet proto tcp \
to port $forwardports -> $lanclient port $lanport

(0) pass out quick on lo0 from any to any
(1) pass in quick on lo0 from any to any

(2) pass in log-all quick on le0
(3) pass in log-all quick on le2
(4) pass out log-all quick on le0
(5) pass out log-all quick on le2

(Rules for external interface not listed: they work, and these
rules, being "quick," catch everything I'm interested in anyway.)

The (numbers) obviously don't appear in the pf.conf file. I
prepended them here for the sake of readability.


Well, traffic between internal interfaces behaves exactly as I would
expect (well, except for the bad checksum message):

rule 2/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108, bad cksum!)
rule 5/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108)
rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170, bad cksum 0!)
rule 4/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170)

"Wonderful," I think, "I can write rules to block asymmetrically
based on interface!" I'm soon sadly disappoionted, as this
statement apparently only applies to traffic traversing the bridge.

Since, for example, I might like to allow access to certain ports on
the OpenBSD box from the LAN, but deny it from the wireless AP.
Obviously, using IP ranges isn't a very effective measure, for
anyone who has breached the AP could easily spoof an IP. Thus, I
want my PF rules to reference the physical interface. But I try
pinging the router from a machine on the LAN, and then from a
machine on the wireless interface, and I get:

LAN machine pings router:

rule 3/0(match): pass in on le2: 192.168.1.9 >
192.168.1.1 : icmp: echo request (id:0f5a seq:0) (ttl 64, id 081)
rule 4/0(match): pass out on le0: 192.168.1.1 >
192.168.1.9 : icmp: echo reply (id:0f5a seq:0) (ttl 255, id 6469)

and:

WLAN machine pings router:

rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo request (id:028b seq:0) (ttl 64, id 48190)
rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo reply (id:028b seq:0) (ttl 255, id 53634)

Incoming traffic TO the OpenBSD box (but not across it) always
matches rules for arrival on le2, and traffic out always matches
rules for departure on le2.

Confirmed if the traffic originates at the OpenBSD box:

Router pings LAN machine:

rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 :
icmp: echo request (id:16ee seq:0) (ttl 255, id 62936)
rule 3/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 :
icmp: echo reply (id:16ee seq:0) (ttl 64, id 3223)

Router pings WLAN machine:

rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo request (id:7ff8 seq:0) (ttl 255, id 60383)
rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo reply (id:7ff8 seq:0) (ttl 64, id 48215)


Okay, then, this has to be some property of the bridge (operating at
a lower level than PF, as it does). But if I can't tell which
interface traffic arrives on at the PF level, CERTAINLY the bridge
rules should be able to tell me. I adjust my bridge and PF rules to
tell pf where the traffic REALLY came from, thus:

/etc/bridgename/bridge0
add le0
add le2
rule pass in on le0 tag t_lan
rule pass in on le2 tag t_wap
up

And adjust my PF rules:

lan = "le0"
wap = "le2"

(0) pass out quick on lo0 from any to any
(1) pass in quick on lo0 from any to any

(2) pass in log-all quick on le0 tagged t_wan keep state
(3) pass in log-all quick on le2 tagged t_lan keep state

(4) pass in log-all quick on le2 tagged t_wan keep state
(5) pass in log-all quick on le0 tagged t_lan keep state

(6) pass out log-all quick on $lan from any to any tagged t_wan keep
state
(7) pass out log-all quick on $lan from any to any tagged t_lan keep
state

(8) pass out log-all quick on $wap from any to any tagged t_wan keep
state
(9) pass out log-all quick on $wap from any to any tagged t_lan keep
state

# in THEORY, none of these should ever fire, as the above
# rules should ALWAYS catch EVERYTHING

(10) pass in log-all quick on le0 keep state
(11) pass in log-all quick on le2 keep state
(12) pass out log-all quick on le0 keep state
(13) pass out log-all quick on le2 keep state


And I try my ICMP experiments again:

LAN machine pinging WLAN machine: echo request matches rules 5 & 9,
as I would expect.

rule 5/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368, bad cksum 0!)
rule 9/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368)
rule 9/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp:
echo reply (id:0f60 seq:0) (ttl 64, id 48246, bad cksum 0!)
rule 5/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f60 seq:0) (ttl 64, id 48246)

Great! The inbound packets on le0 are tagged properly, and
remain tagged properly as they pass out le2 to their destination.

Echo reply matches the SAME rules. I might have expected rules
4 & 6 to catch the echo reply, but I presume that "keep state"
makes the "response" traffic pass by the same rules that the
connection-establishing message matched... Brilliant!

This means I can allow ssh connections originating FROM the LAN
TO the AP to be established, and their statefulness will allow the
return traffic, but I can BLOCK ssh connections originating FROM the
AP TO the LAN!

My elation didn't last long, however...

WLAN machine pings LAN machine:

rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 :
icmp: echo request (id:028e seq:0) (ttl 64, id 48262, bad cksum 0!)
rule 12/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo request (id:028e seq:0) (ttl 64, id 48262)
rule 12/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp:
echo reply (id:028e seq:0) (ttl 64, id 3410, bad cksum 0!)
rule 11/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp:
echo reply (id:028e seq:0) (ttl 64, id 3410)

Absolutely none of the tagged rules, which ought to be
exhaustive, match. This tells me that the bridge is only
tagging traffic that appears on le0, not on le2.

brconfig (8) says:

Rules are processed in the order in which they were added to
the interface, and the first rule matched takes the action
(block or pass) and, if given, the tag of the rule. If no
source or destination address is specified, the rule will
match all frames (good for creating a catchall policy).

I'm not really sure what's going on here. I mean, if this is
literal, then my first bridge rule ALWAYS applies, since I've
specified no hardware addresses at al. But that would mean that ALL
packets would be tagged "t_lan." However, what I find is that
these packets were apparently not tagged AT ALL.

Well, at least the interfaces that the traffic appears on are
correct.


LAN machine pings router:

rule 11/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 :
icmp: echo request (id:0f5f seq:0) (ttl 64, id 3350)
rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 :
icmp: echo reply (id:0f5f seq:0) (ttl 255, id 8988)

WLAN machine pings router:

rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo request (id:028d seq:0) (ttl 64, id 48254)
rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo reply (id:028d seq:0) (ttl 255, id 10230)

Drat.

I cannot quite express how frustrating this is. Not only did I
write a rule that would pass in traffic on le2 matching the correct
"t_wan" tag (rule 4), I even wrote a rule that would pass in traffic
on le2 matching the WRONG "t_lan" tag (rule 3). Likewise for le0
(correct: rule 5; incorrect: rule 2).

Traffic from the LAN should be tagged "t_lan" by the bridge rule,
and should hit PF rule 5. Even if it gets tagged "t_wan" by bridge
(in the event that I'm misunderstanding bridge rules), it should
pass rule 2. What I find is that it's passing rule 11! Which tells
me that bridge is not actually tagging inbound traffic.

Well, that might be expected in the LAN->router and WLAN->router
cases, since the traffic was not destined for either of the bridged
interfaces, the bridge may not touch it at all, and thus wouldn't
tag the traffic.

But it should NOT be the case for WLAN->LAN traffic, which it
appears to be.

If I `brconfig bridge0 down,` traffic always appears on the correct
interface. (But then, obviously, things like 224.x.x.x broadcasts
don't cross from le0 to le2, and vice versa.)

This seems wrong. If the bridge "touches" a packet, and changes the
interface pf sees it on, then it should be able to tag it, so that
pf can be told which interface it came from. If the bridge won't
tag an inbound-only packet, then it shouldn't change the interface
that pf sees. Perhaps this is simply a shortcoming in design, and
what I want to do is not possible at all.

So, here are the problems:

1. bridge rules aren't tagging packets coming in on le2.
2. bridge incorrectly marks traffic INBOUND to the OpenBSD box
as coming from le2, even if it came in on le0.
3. bridge incorrectly marks traffic OUTBOUND from the OpenBSF
box as going to le0, even if it physically appears only on le2.

And here are the questions:

1. What's wrong with my bridge rules, that le2 traffic isn't
tagged at all?

2. What can I do to preserve physical interface information on
the bridged interface for pf?

What am I doing wrong? Is there any way to solve this problem? If
bridge doesn't touch inbound traffic, then PF should see it
appearing on the correct interface. If it does, then it should be
able to tag it. This dichotomy does not make sense.

I would think that there should be SOME way to preserve the physical
interface info for pf. After all, this is security, and the MOST
important bit of information we have about a packet is what physical
device received it:

IP, TCP, UDP, and ICMP headers can be spoofed.

Link-level headers can be spoofed.

Where the wire is plugged in at the back of the box cannot.


Thanks,
JMF

--
MKS

Nathan Kennedy wrote:

(Sorry about the double-post. Machine wasn't logged in as me.)


>> I've tried to use bridge rules, but have had even more problems with
>> those, as applying the following to my bridgename.bridge0:

>> rule pass in on le0 tag t_lan
>> rule pass in on le2 tag t_wap

> I'm not sure what you're doing there... I didn't put any rules in
> my bridgename.bridge0, just "add le0 add le2 up". I put all the
> firewall rules in pf.conf. I didn't use any tagging.

Sorry, I could have been clearer. I explain in more detail later,
but I want different rules for clients on one interface of the
bridge than for clients on the other. Since PF alone seems to get
confused as to which physical interface traffic goes in and out on
for the bridge, I was hoping to have the bridge tell it.


>> inbound traffic from le0 to le2, and vice versa, always matches rules
>> for the correct interfaces inbound traffic from le0 AND le2 to the
>> router always appears to match rules for le2 outbound traffic from
>> the router to le0 AND le2 always appears to match rules for le0

>> Why? Can I depend on this to always be the case? How do I know
>> which interface it will pick for the 'outbound,' and which it
>> will pick for the 'inbound'?

> You can pick this yourself.

How? Right now, it picks the "correct" interface for traffic
traversing the bridge, and picks le2 as the inbound interface for
all traffic NOT crossing the bridge, and le0 as the outbound for all
traffic NOT originating from the bridge. What did I do to "pick"
this configuration?


> It doesn't matter which way you do it. You MUST specify filter
> rules in both directions for both interfaces in your pf.conf
> file!

If you see my logs later, I'm not sure why I "must." Stateless
traffic appears to require rules for both directions, both
interfaces, but stateful traffic only seems to log against two
rules -- those that established the connection.

Of course, two rules only allows connections to be opened from one
specific side of the bridge to the other, but then, that's one of
the things I'm after. :-)

> On one interface you'll want to pass in and out everything. And
> you do the actual firewall rules on the other interface. It
> doesn't matter which interface you do the filtering on, it's
> symmetric.

> ----------------------
> | |
> out<-le0->in BRIDGE in<-le2->out
> | |
> ----------------------

Yes, I get that. Mine is a little more complex, in that I'm asking
two additional functions out of the device. Most of the literature,
it seems, focuses on the notion of a box that acts as a
"transparent" filtering bridge.

For my application, I want the two networks on either side of the
bridge to have this transparency when communicating between each
other, but I also have the OpenBSD box acting as NAT on a third
interface, DNS to the two bridged interfaces (so it must have an
IP), and to be accessible for certain other services (e.g. SSH) to
only ONE of the bridged interfaces. Thus, I DON'T want to just pass
everything on, say, le2, and filter only on le0, because there is
traffic that is not only crossing the bridge, but going TO the bridge.

-------------------------
| |
| /->}BRIDGE}<-\ |
| | | |
| v v |
out<-le0-->in--->DNS<---in<--le2->out
| ^ | ^ |
| | +->SSH | |
| | | | |
| | \->HTTP | |
| | v |
| \-----FWD<--------NAT<-le1->out
| |
-------------------------

Noting that the box will filter traffic differently on le0 and
le2. (e.g. le2 should get DNS from the box, but will not be able to
SSH into it, or into machines on le0, but the box and machines on
le0 will be able to SSH into machines on le2.)

As I said, if I use stateful rules, it appears this can be done
fairly easily. My problem isn't that the rules aren't can't do what
I want, in theory, it's that the pf/bridge combination seems unable
to determine the physical interfaces for traffic to and from the
OpenBSD machine itself.

When traffic bound for the OpenBSD box hits pf, it appears as though
it comes from le2. That is, when I ping OpenBSD from a machine on
le0, it matches rules for "in on le2."


> Can you post your entire bridgename.bridge0 and pf.conf files, as
> well as explain what exactly the bridge's function is?

Certainly. I do so in just a paragraph or so... actually, I've
stripped out comments, and rules relating to blocking on my third
(NAT) interface, as they don't come into play.


> If it's just a bridge between two networks and you don't want to
> filter then you shouldn't need any rules or tagging at all, just
> pass in and out everything. By default it should learn what's
> where and be well-behaved, assuming there's no cycles in your
> network.

I do want to filter, I want to filter asymmetrically, and it IS
MOSTLY well-behaved. As for "cycles" in the network, I haven't
any. There's a switch on le0, a hub on le2, and only simple clients
with a single interface are plugged into either.

As to what I'm trying to accomplish, this is for adding a wireless
access point to a basic home setup with a cable/DSL connection. I
don't want to hang the AP in a DMZ, because (a) I want to provide
the same level of "protection" to the AP as I do to the LAN, and (b)
because there are certain services (e.g. iTunes autodiscovery) that
I want to be functional between the LAN and AP.


# OpenBSD box with 3 interfaces
#
# le0 is the internal wired LAN
# le1 is the external internet
# le2 is the internal wireless AP
#
# Goals:
#
# Perform NAT for both internal networks to the outside world
# Bridge LAN and wireless AP to provide certain "broadcast-based"
# services (Rendezvous/iTunes, etc.) the impression that they
# span the network
# Provide DNS service to LAN and wireless
# Provide "full" internet functionality to LAN and wireless
# Provice "full" access to machines on the wireless from the LAN
# Provide "limited" services on LAN machines to wireless clients
# "Protect" the router, LAN, and AP from the internet
# "Protect" the router and LAN from the wireless

As you can see, I want the two internal interfaces to be asymmetrical
with respect to what they can do to each other, and with what
services they can access on the OpenBSD box. Additionally, simply
using two separate subnets won't work, because some services use
broadcast messages to discover other nodes. For example, iTunes
broadcasts UDP packets on 224.x.x.x.

Of course, I started with my "best guess" ruleset, and quickly
learned that I had no idea what I was really doing. :-)

So, I decided to use some simple rulesets, with logging, to get a
better understanding of how this is all working. Start by passing
and logging everything, and write rules accordingly to pick off the
traffic that I *do* want to pass, then drop the pass-all rules in
favor of block-all. Sounds reasonable, right?

I'll spare you the miserable failure of my first attempt, and jump
right into the testing setups:

First, I tried:

/etc/hostname.le0
inet 192.168.1.1 255.255.255.0 NONE

/etc/hostname.le1
dhcp NONE NONE NONE

/etc/hostname.le2
up

/etc/bridgename/bridge0
add le0
add le2
up

with PF rules (I've not listed my tables or altq rules for the
external interface, as they do not come into play here):

set optimization normal
set block-policy return

scrub in all fragment reassemble
scrub out on $ext random-id

nat on $ext from 192.168.1.0/24 to any -> $ext

rdr pass on $ext inet proto tcp \
to port $forwardports -> $lanclient port $lanport

(0) pass out quick on lo0 from any to any
(1) pass in quick on lo0 from any to any

(2) pass in log-all quick on le0
(3) pass in log-all quick on le2
(4) pass out log-all quick on le0
(5) pass out log-all quick on le2

(Rules for external interface not listed: they work, and these
rules, being "quick," catch everything I'm interested in anyway.)

The (numbers) obviously don't appear in the pf.conf file. I
prepended them here for the sake of readability.


Well, traffic between internal interfaces behaves exactly as I would
expect (well, except for the bad checksum message):

rule 2/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108, bad cksum!)
rule 5/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108)
rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170, bad cksum 0!)
rule 4/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170)

"Wonderful," I think, "I can write rules to block asymmetrically
based on interface!" I'm soon sadly disappoionted, as this
statement apparently only applies to traffic traversing the bridge.

Since, for example, I might like to allow access to certain ports on
the OpenBSD box from the LAN, but deny it from the wireless AP.
Obviously, using IP ranges isn't a very effective measure, for
anyone who has breached the AP could easily spoof an IP. Thus, I
want my PF rules to reference the physical interface. But I try
pinging the router from a machine on the LAN, and then from a
machine on the wireless interface, and I get:

LAN machine pings router:

rule 3/0(match): pass in on le2: 192.168.1.9 >
192.168.1.1 : icmp: echo request (id:0f5a seq:0) (ttl 64, id 081)
rule 4/0(match): pass out on le0: 192.168.1.1 >
192.168.1.9 : icmp: echo reply (id:0f5a seq:0) (ttl 255, id 6469)

and:

WLAN machine pings router:

rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo request (id:028b seq:0) (ttl 64, id 48190)
rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo reply (id:028b seq:0) (ttl 255, id 53634)

Incoming traffic TO the OpenBSD box (but not across it) always
matches rules for arrival on le2, and traffic out always matches
rules for departure on le2.

Confirmed if the traffic originates at the OpenBSD box:

Router pings LAN machine:

rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 :
icmp: echo request (id:16ee seq:0) (ttl 255, id 62936)
rule 3/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 :
icmp: echo reply (id:16ee seq:0) (ttl 64, id 3223)

Router pings WLAN machine:

rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo request (id:7ff8 seq:0) (ttl 255, id 60383)
rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo reply (id:7ff8 seq:0) (ttl 64, id 48215)


Okay, then, this has to be some property of the bridge (operating at
a lower level than PF, as it does). But if I can't tell which
interface traffic arrives on at the PF level, CERTAINLY the bridge
rules should be able to tell me. I adjust my bridge and PF rules to
tell pf where the traffic REALLY came from, thus:

/etc/bridgename/bridge0
add le0
add le2
rule pass in on le0 tag t_lan
rule pass in on le2 tag t_wap
up

And adjust my PF rules:

lan = "le0"
wap = "le2"

(0) pass out quick on lo0 from any to any
(1) pass in quick on lo0 from any to any

(2) pass in log-all quick on le0 tagged t_wan keep state
(3) pass in log-all quick on le2 tagged t_lan keep state

(4) pass in log-all quick on le2 tagged t_wan keep state
(5) pass in log-all quick on le0 tagged t_lan keep state

(6) pass out log-all quick on $lan from any to any tagged t_wan keep
state
(7) pass out log-all quick on $lan from any to any tagged t_lan keep
state

(8) pass out log-all quick on $wap from any to any tagged t_wan keep
state
(9) pass out log-all quick on $wap from any to any tagged t_lan keep
state

# in THEORY, none of these should ever fire, as the above
# rules should ALWAYS catch EVERYTHING

(10) pass in log-all quick on le0 keep state
(11) pass in log-all quick on le2 keep state
(12) pass out log-all quick on le0 keep state
(13) pass out log-all quick on le2 keep state


And I try my ICMP experiments again:

LAN machine pinging WLAN machine: echo request matches rules 5 & 9,
as I would expect.

rule 5/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368, bad cksum 0!)
rule 9/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130:
icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368)
rule 9/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp:
echo reply (id:0f60 seq:0) (ttl 64, id 48246, bad cksum 0!)
rule 5/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo reply (id:0f60 seq:0) (ttl 64, id 48246)

Great! The inbound packets on le0 are tagged properly, and
remain tagged properly as they pass out le2 to their destination.

Echo reply matches the SAME rules. I might have expected rules
4 & 6 to catch the echo reply, but I presume that "keep state"
makes the "response" traffic pass by the same rules that the
connection-establishing message matched... Brilliant!

This means I can allow ssh connections originating FROM the LAN
TO the AP to be established, and their statefulness will allow the
return traffic, but I can BLOCK ssh connections originating FROM the
AP TO the LAN!

My elation didn't last long, however...

WLAN machine pings LAN machine:

rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 :
icmp: echo request (id:028e seq:0) (ttl 64, id 48262, bad cksum 0!)
rule 12/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 :
icmp: echo request (id:028e seq:0) (ttl 64, id 48262)
rule 12/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp:
echo reply (id:028e seq:0) (ttl 64, id 3410, bad cksum 0!)
rule 11/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp:
echo reply (id:028e seq:0) (ttl 64, id 3410)

Absolutely none of the tagged rules, which ought to be
exhaustive, match. This tells me that the bridge is only
tagging traffic that appears on le0, not on le2.

brconfig (8) says:

Rules are processed in the order in which they were added to
the interface, and the first rule matched takes the action
(block or pass) and, if given, the tag of the rule. If no
source or destination address is specified, the rule will
match all frames (good for creating a catchall policy).

I'm not really sure what's going on here. I mean, if this is
literal, then my first bridge rule ALWAYS applies, since I've
specified no hardware addresses at al. But that would mean that ALL
packets would be tagged "t_lan." However, what I find is that
these packets were apparently not tagged AT ALL.

Well, at least the interfaces that the traffic appears on are
correct.


LAN machine pings router:

rule 11/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 :
icmp: echo request (id:0f5f seq:0) (ttl 64, id 3350)
rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 :
icmp: echo reply (id:0f5f seq:0) (ttl 255, id 8988)

WLAN machine pings router:

rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 :
icmp: echo request (id:028d seq:0) (ttl 64, id 48254)
rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130:
icmp: echo reply (id:028d seq:0) (ttl 255, id 10230)

Drat.

I cannot quite express how frustrating this is. Not only did I
write a rule that would pass in traffic on le2 matching the correct
"t_wan" tag (rule 4), I even wrote a rule that would pass in traffic
on le2 matching the WRONG "t_lan" tag (rule 3). Likewise for le0
(correct: rule 5; incorrect: rule 2).

Traffic from the LAN should be tagged "t_lan" by the bridge rule,
and should hit PF rule 5. Even if it gets tagged "t_wan" by bridge
(in the event that I'm misunderstanding bridge rules), it should
pass rule 2. What I find is that it's passing rule 11! Which tells
me that bridge is not actually tagging inbound traffic.

Well, that might be expected in the LAN->router and WLAN->router
cases, since the traffic was not destined for either of the bridged
interfaces, the bridge may not touch it at all, and thus wouldn't
tag the traffic.

But it should NOT be the case for WLAN->LAN traffic, which it
appears to be.

If I `brconfig bridge0 down,` traffic always appears on the correct
interface. (But then, obviously, things like 224.x.x.x broadcasts
don't cross from le0 to le2, and vice versa.)

This seems wrong. If the bridge "touches" a packet, and changes the
interface pf sees it on, then it should be able to tag it, so that
pf can be told which interface it came from. If the bridge won't
tag an inbound-only packet, then it shouldn't change the interface
that pf sees. Perhaps this is simply a shortcoming in design, and
what I want to do is not possible at all.

So, here are the problems:

1. bridge rules aren't tagging packets coming in on le2.
2. bridge incorrectly marks traffic INBOUND to the OpenBSD box
as coming from le2, even if it came in on le0.
3. bridge incorrectly marks traffic OUTBOUND from the OpenBSF
box as going to le0, even if it physically appears only on le2.

And here are the questions:

1. What's wrong with my bridge rules, that le2 traffic isn't
tagged at all?

2. What can I do to preserve physical interface information on
the bridged interface for pf?

What am I doing wrong? Is there any way to solve this problem? If
bridge doesn't touch inbound traffic, then PF should see it
appearing on the correct interface. If it does, then it should be
able to tag it. This dichotomy does not make sense.

I would think that there should be SOME way to preserve the physical
interface info for pf. After all, this is security, and the MOST
important bit of information we have about a packet is what physical
device received it:

IP, TCP, UDP, and ICMP headers can be spoofed.

Link-level headers can be spoofed.

Where the wire is plugged in at the back of the box cannot.


Thanks,
JMF

--
JMF