{"id":278,"date":"2017-07-24T17:13:21","date_gmt":"2017-07-24T15:13:21","guid":{"rendered":"http:\/\/blog.le-vert.net\/?p=278"},"modified":"2017-07-24T17:14:13","modified_gmt":"2017-07-24T15:14:13","slug":"policy-based-routing-pbr-with-shorewall-to-migrate-a-server","status":"publish","type":"post","link":"https:\/\/blog.le-vert.net\/?p=278","title":{"rendered":"Policy Based Routing (PBR) with Shorewall to migrate a server"},"content":{"rendered":"<div class=\"twttr_buttons\"><div class=\"twttr_twitter\">\n\t\t\t\t\t<a href=\"http:\/\/twitter.com\/share?text=Policy+Based+Routing+%28PBR%29+with+Shorewall+to+migrate+a+server\" class=\"twitter-share-button\" data-via=\"\" data-hashtags=\"\"  data-size=\"default\" data-url=\"https:\/\/blog.le-vert.net\/?p=278\"  data-related=\"\" target=\"_blank\">Tweet<\/a>\n\t\t\t\t<\/div><\/div><p>Hey,<\/p>\n<p>Today I&#8217;m doing pure sysadmin work and I&#8217;ve been asked to migrate several servers from an obsolete IP range 192.168.x to 10.x. Things were quite easy until I reached the internal mail server that can be used by hundreds field hardware as a relay server. Everybody is supposed to use DNS entry but I won&#8217;t trust that.<\/p>\n<p>So my idea is to switch eth0 to the new network and keep a new eth1 in the old one to keep the service working and be able to log what&#8217;s using the obsolete address.<br \/>\nThere&#8217;s just a little problem: if my default gateway is on eth0, any packet entering eth1 from a routed network (would work for what&#8217;s connected in the legacy local network) will be answered using the default gateway on eth0. That&#8217;s asymmetric routing and that just doesn&#8217;t work.<\/p>\n<p>Okay, so how do I solve that ? With Shorewall of course ! The idea if to tag any packet entering eth1 with a different mark than the ones coming throught eth0 and provide different routing table for each mark. I&#8217;ll do this on CentOS today but it should be basically the same for any Linux system. Shorewall is usually available everywhere but you can try doing this by hand with &#8220;ip&#8221; and &#8220;iptables&#8221;. Looks like a lot of pain to me, though.<br \/>\nHaving both address routed and working is a nice step but it&#8217;s pretty useless if I have no way to find out who&#8217;s using the obsolete address so we&#8217;ll use Shorewall to log these access and create a specific rsyslog\/logrotate configuration to get a dedicated log.<\/p>\n<p>First, we&#8217;ll change the network configuration to have both interfaces up with a default gateway only on the first interface (connected to the new network). The gateway will be later overridden by Shorewall but it&#8217;s always saner to have a default configuration working, even with limited feature.<\/p>\n<p>So make sure to create proper <code>ifcfg-eth0<\/code> and <code>ifcfg-eth1<\/code> in <code>\/etc\/sysconfig\/network-scripts<\/code> and make sure to have only <code>GATEWAY<\/code> defined on the new network. You should also make sure that the server is reachable on its new address and reachable on the old address with a machine directly connected to the legacy network.<\/p>\n<p>Let&#8217;s continue with a very basic Shorewall configuration. <code>yum -y install shorewall<\/code> and then make sure to have the three following files in <code>\/etc\/shorewall<\/code>:<\/p>\n<ul>\n<li><strong>interfaces<\/strong> &#8211; List of network adapter handled by Shorewall<\/li>\n<li><strong>policy<\/strong> &#8211; Default firewall policies between each zone<\/li>\n<li><strong>providers<\/strong> &#8211; This one is PBR specific, we&#8217;ll use this to mark packets<\/li>\n<li><strong>rules<\/strong> &#8211; Overrides default policies with port\/host rules<\/li>\n<li><strong>shorewall.conf<\/strong> &#8211; Global settings<\/li>\n<li><strong>zones<\/strong> &#8211; Map interfaces to firewall zones<\/li>\n<\/li>\n<p>If you miss one, copy it from <code>\/usr\/share\/shorewall\/configfiles\/<\/code><\/p>\n<p>So let&#8217;s do a few adjustments in <code>shorewall.conf<\/code> first:<br \/>\n<code><br \/>\n<strong>IP_FORWARDING=No<\/strong> (No this machine SHOULD never be used a gateway between legacy and new network, we're not here to create security flows ;-))<br \/>\n<strong>DISABLE_IPV6=Yes<\/strong> (Sadly, there's no IPv6 here so it's better to let Shorewall4 kill the whole stack)<br \/>\n<strong>LOGTAGONLY=Yes<\/strong> (Change the way Shorewall generate log prefix, otherwise ours will be too long and get shortened)<br \/>\n<\/code><\/p>\n<p>Now defines the interfaces in <code>rules<\/code>:<\/p>\n<pre>\r\nloc   eth0   tcpflags,logmartians,nosmurfs,sourceroute=0\r\nold   eth1   tcpflags,logmartians,nosmurfs,sourceroute=0\r\n<\/pre>\n<p>And map them to IPv4 <code>zones<\/code>:<\/p>\n<pre>\r\nfw      firewall\r\nloc     ipv4\r\nold     ipv4\r\n<\/pre>\n<p>fw is a default zone meaning &#8220;myself&#8221;.<\/p>\n<p>And we create a default <code>policy<\/code> allowing the machine itself to reach legacy and new network zones and blocking any incoming packets.<\/p>\n<pre>\r\nfw              loc,old         ACCEPT\r\nall             all             DROP            info\r\n<\/pre>\n<p>Finally we&#8217;ll add a set of default <code>rules<\/code> to be at least, to SSH the server again<\/p>\n<pre>\r\nPing(ACCEPT)    loc         fw\r\nSSH(ACCEPT)     loc         fw\r\n<\/pre>\n<p>Just like in policy file, you can use loc,old if you want to permit ping and SSH from the old network too.<\/p>\n<p>I&#8217;ll also add a few <code>rules<\/code> to permit mail related services from the new zone:<\/p>\n<pre>\r\nSMTP(ACCEPT)            loc     fw\r\nSubmission(ACCEPT)      loc     fw\r\nIMAP(ACCEPT)            loc     fw\r\n<\/pre>\n<p>Okay, now we can enable and start Shorewall.<\/p>\n<pre>\r\nsystemctl enable shorewall\r\nsystemctl start shorewall\r\n<\/pre>\n<p>Now we&#8217;ll ask Shorewall to mark packets differently according to the incoming interface. This will be done in <code>providers<\/code> file.<\/p>\n<pre>\r\nNEW  1  1  -  eth0  10.0.0.254     track\r\nOLD  2  2  -  eth1  192.168.0.254  track\r\n<\/pre>\n<p>Last column is the gateway to use on each network.<\/p>\n<p>Let&#8217;s permit mail-related traffic from the legacy network but ask Shorewall to log these packets. Add the following in <code>rules<\/code> file:<\/p>\n<pre>\r\nSMTP(ACCEPT):info:MailMigration        old  fw\r\nSubmission(ACCEPT):info:MailMigration  old  fw\r\nIMAP(ACCEPT):info:MailMigration        old  fw\r\n<\/pre>\n<p>Reload Shorewall and try to telnet tcp\/25 <strong>from a routed network<\/strong>, both IPs are now working !<\/p>\n<pre>systemctl reload shorewall<\/pre>\n<p>If you check <code>\/var\/log\/messages<\/code> you will see log like:<br \/>\n<code>Jul 24 16:55:39 mailsrv kernel: Shorewall:MailMigration:ACCE IN=eth1 OUT= MAC=XXX SRC=192.168.55.4 DST=192.168.0.10 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=22405 DF PROTO=TCP SPT=39474 DPT=25 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x2<\/code><\/p>\n<p>You can also check your routing tables with <strong>ip route show table<\/strong>.<br \/>\n<strong>ip route show table main<\/strong> shows no more default gateway<br \/>\n<strong>ip route show table 1<\/strong> shows local route for eth0 network and default gateway of the new network. It&#8217;ll be used for packet tagged as 1.<br \/>\n<strong>ip route show table 2<\/strong> shows local route for eth1 network and default gateway of the legacy network. It&#8217;ll be used for packet tagged as 2 (Note the log above with MARK=0x2).<\/p>\n<p>Your server is now completely accessible from both networks and you can easily monitor the log file to find clients still using the legacy address. But we can make it a lot easier by asking rsyslog to create a separate log file with these specific messages:<\/p>\n<p>Create <code>\/etc\/rsyslog.d\/mailmigration.conf<\/code> with the following content:<\/p>\n<pre>\r\n$FileOwner root\r\n$FileGroup root\r\n$Umask 0000\r\n$FileCreateMode 0644\r\n:msg, startswith, \"Shorewall:MailMigration:\" \/var\/log\/mailmigration.log\r\n& stop\r\n<\/pre>\n<p>And the associated logrotate <code>\/etc\/logrotate.d\/mailmigration<\/code> file to avoid having a single never ending file:<\/p>\n<pre>\r\n\r\n\/var\/log\/mailmigration.log\r\n{\r\n    missingok\r\n    sharedscripts\r\n    postrotate\r\n        \/bin\/kill -HUP `cat \/var\/run\/syslogd.pid 2> \/dev\/null` 2> \/dev\/null || true\r\n    endscript\r\n}\r\n<\/pre>\n<p>If you want to go further for a more automated way of handling this, I&#8217;d definitely suggest having a look at <a href=\"http:\/\/www.rsyslog.com\/doc\/v8-stable\/configuration\/modules\/omamqp1.html\">Rsyslog AMQP module<\/a> to publish event to a RabbitMQ and write a quick Python consumer to parse and notify &#8220;Someone&#8221; (may I suggest calling some API to create an internal support ticket ?) using <a href=\"https:\/\/www.rabbitmq.com\/tutorials\/tutorial-two-python.html\">Pika<\/a>. The &#8220;worker.py&#8221; file should be enough for testing, just try\/except\/ch.basic_nack your handler so the message goes back in queue in case of failure.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hey, Today I&#8217;m doing pure sysadmin work and I&#8217;ve been asked to migrate several servers from an obsolete IP range 192.168.x to 10.x. Things were quite easy until I reached the internal mail server that can be used by hundreds &hellip; <a href=\"https:\/\/blog.le-vert.net\/?p=278\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/posts\/278"}],"collection":[{"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=278"}],"version-history":[{"count":12,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/posts\/278\/revisions"}],"predecessor-version":[{"id":290,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=\/wp\/v2\/posts\/278\/revisions\/290"}],"wp:attachment":[{"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=278"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=278"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.le-vert.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=278"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}