StrongSwan: Encryption not supported

Posted on 2019/12/04 by Le_Vert

Introduction

IPSec always sounded like a nightmare to me, at least, as a long time user of OpenVPN I never understood why it is so complicated.

But hello GCE, AWS and customer asking me to join on-premise networks to their cloud provider. There’s no alternative here, but IPSec. If you don’t want to do it on Cisco (or assimilated devices) there’s StrongSwan on Linux but there’s a huge pitfall and I wanted to write about it.

Usually, you start having cipher negotiation issue and StrongSwan logging is to say the least, not helpful.

Debian tricked me

Today I was connecting a Google Cloud to a Debian-based gateway with StrongSwan and as expected I got cipher issue:

ipsec up gce-vpn

 initiating IKE_SA gce-vpn[282] to 1.2.3.4
 generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
 sending packet: from 127.0.0.1[500] to 1.2.3.4[500] (874 bytes)
 received packet: from 1.2.3.4[500] to 127.0.0.1[500] (712 bytes)
 parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]
 selected proposal: IKE:AES_GCM_16_256/PRF_HMAC_SHA2_512/MODP_4096
 ENCRYPTION_ALGORITHM AES_GCM_16 (key size 256) not supported!
 key derivation failed
 establishing connection 'gce-vpn' failed

ipsec up gce-vpn

initiating IKE_SA gce-vpn[282] to 1.2.3.4

generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]

sending packet: from 127.0.0.1[500] to 1.2.3.4[500] (874 bytes)

received packet: from 1.2.3.4[500] to 127.0.0.1[500] (712 bytes)

parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]

selected proposal: IKE:AES_GCM_16_256/PRF_HMAC_SHA2_512/MODP_4096

ENCRYPTION_ALGORITHM AES_GCM_16 (key size 256) not supported!

key derivation failed

establishing connection 'gce-vpn' failed

So the first question was, is AES_GCM_16 not supported on my side or on Google side ? Cryptic messages did not help but I assumed it was on my side as when it comes from the other side the message is usually “NO_PROPOSAL_CHOSEN”.

How do I check supported ciphers ?

ipsec listalgs

List of registered IKE algorithms:
 encryption: AES_CBC[aes] RC2_CBC[rc2]
   integrity:  AES_XCBC_96[xcbc] HMAC_SHA1_96[hmac] HMAC_SHA1_128[hmac] HMAC_SHA1_160[hmac] HMAC_MD5_96[hmac]
               HMAC_MD5_128[hmac] HMAC_SHA2_256_128[hmac] HMAC_SHA2_256_256[hmac] HMAC_SHA2_384_192[hmac]
               HMAC_SHA2_384_384[hmac] HMAC_SHA2_512_256[hmac] HMAC_SHA2_512_512[hmac]
   aead:      
   hasher:     HASH_SHA1[sha1] HASH_SHA2_224[sha2] HASH_SHA2_256[sha2] HASH_SHA2_384[sha2] HASH_SHA2_512[sha2]
               HASH_MD5[md5]
   prf:        PRF_KEYED_SHA1[sha1] PRF_FIPS_SHA1_160[fips-prf] PRF_AES128_XCBC[xcbc] PRF_HMAC_SHA1[hmac]
               PRF_HMAC_MD5[hmac] PRF_HMAC_SHA2_256[hmac] PRF_HMAC_SHA2_384[hmac] PRF_HMAC_SHA2_512[hmac]
   xof:        XOF_MGF1_SHA1[mgf1] XOF_MGF1_SHA224[mgf1] XOF_MGF1_SHA256[mgf1] XOF_MGF1_SHA384[mgf1]
               XOF_MGF1_SHA512[mgf1]
   dh-group:   MODP_3072[gmp] MODP_4096[gmp] MODP_6144[gmp] MODP_8192[gmp] MODP_2048[gmp] MODP_2048_224[gmp]
               MODP_2048_256[gmp] MODP_1536[gmp] MODP_1024[gmp] MODP_1024_160[gmp] MODP_768[gmp] MODP_CUSTOM[gmp]
   random-gen: RNG_STRONG[random] RNG_TRUE[random]
   nonce-gen:  [nonce]

ipsec listalgs

List of registered IKE algorithms:

encryption: AES_CBC[aes] RC2_CBC[rc2]

integrity: AES_XCBC_96[xcbc] HMAC_SHA1_96[hmac] HMAC_SHA1_128[hmac] HMAC_SHA1_160[hmac] HMAC_MD5_96[hmac]

HMAC_MD5_128[hmac] HMAC_SHA2_256_128[hmac] HMAC_SHA2_256_256[hmac] HMAC_SHA2_384_192[hmac]

HMAC_SHA2_384_384[hmac] HMAC_SHA2_512_256[hmac] HMAC_SHA2_512_512[hmac]

aead:

hasher: HASH_SHA1[sha1] HASH_SHA2_224[sha2] HASH_SHA2_256[sha2] HASH_SHA2_384[sha2] HASH_SHA2_512[sha2]

HASH_MD5[md5]

prf: PRF_KEYED_SHA1[sha1] PRF_FIPS_SHA1_160[fips-prf] PRF_AES128_XCBC[xcbc] PRF_HMAC_SHA1[hmac]

PRF_HMAC_MD5[hmac] PRF_HMAC_SHA2_256[hmac] PRF_HMAC_SHA2_384[hmac] PRF_HMAC_SHA2_512[hmac]

xof: XOF_MGF1_SHA1[mgf1] XOF_MGF1_SHA224[mgf1] XOF_MGF1_SHA256[mgf1] XOF_MGF1_SHA384[mgf1]

XOF_MGF1_SHA512[mgf1]

dh-group: MODP_3072[gmp] MODP_4096[gmp] MODP_6144[gmp] MODP_8192[gmp] MODP_2048[gmp] MODP_2048_224[gmp]

MODP_2048_256[gmp] MODP_1536[gmp] MODP_1024[gmp] MODP_1024_160[gmp] MODP_768[gmp] MODP_CUSTOM[gmp]

random-gen: RNG_STRONG[random] RNG_TRUE[random]

nonce-gen: [nonce]

Indeed, I cannot see anything related to GCM, looks being the root of my issue.

Investigations and resolution

I wanted to verify first if this cipher is supposed to be supported by StrongWan and found the answer here:

https://wiki.strongswan.org/projects/strongswan/wiki/IKEv2CipherSuites

It needs “aes” and “gcm” plugins, so let’s read more about StrongSwan plugins

https://wiki.strongswan.org/projects/strongswan/wiki/PluginList

Ok that’s interresting. Aes should be ok, but gcm is disabled by default. I need to check what Debian did here, and verify the gcm plugin is enabled.

Debian developers usually store their packaging files on Debian’s internal public GitLab server which is called Salsa. Strongswan packaging is hosted here, so it was easy to check:

https://salsa.debian.org/debian/strongswan/blob/debian/master/debian/rules

debian/rules file is in charge of building the package, so this is what you want to check for options passed ton autotools configure script. Here we can clearly see gcm plugin is explicitly enabled.

I also checked debian/changelog to be sure it was already enabled in the stable version of the package which is available on Debian Buster, and yes, it was…

So why do I missed it ?

I check installed strongswan packages and found *libstrongswan*, let’s see it content

dpkg -L libstrongswan

[...]
/usr/lib/ipsec/plugins/libstrongswan-aes.so
/usr/lib/ipsec/plugins/libstrongswan-attr.so
/usr/lib/ipsec/plugins/libstrongswan-constraints.so
[...]

dpkg -L libstrongswan

[...]

/usr/lib/ipsec/plugins/libstrongswan-aes.so

/usr/lib/ipsec/plugins/libstrongswan-attr.so

/usr/lib/ipsec/plugins/libstrongswan-constraints.so

[...]

This is very interesting ! The aes plugin file is here, but no gcm one. Let’s check if it’s available (i’ll just guess its name) anywhere in the Debian archive ?

apt install apt-file
apt-file update
apt-file search libstrongswan-gcm.so

libstrongswan-standard-plugins: /usr/lib/ipsec/plugins/libstrongswan-gcm.so

apt install apt-file

apt-file update

apt-file search libstrongswan-gcm.so

libstrongswan-standard-plugins: /usr/lib/ipsec/plugins/libstrongswan-gcm.so

I felt stupid here… This is quite common in Debian to separate the “standard” files from the “minimal” ones.

After installing the package and restarting strongswan the list of supported cipher is quite different:

apt install libstrongswan-standard-plugins
systemctl restart strongswan
ipsec listalgs

List of registered IKE algorithms:
 encryption: AES_CBC[aesni] AES_CTR[aesni] RC2_CBC[rc2] 3DES_CBC[openssl] CAMELLIA_CBC[openssl] CAST_CBC[openssl]
               BLOWFISH_CBC[openssl] DES_CBC[openssl] DES_ECB[openssl] NULL[openssl]
   integrity:  AES_XCBC_96[aesni] AES_CMAC_96[aesni] HMAC_MD5_96[openssl] HMAC_MD5_128[openssl] HMAC_SHA1_96[openssl]
               HMAC_SHA1_128[openssl] HMAC_SHA1_160[openssl] HMAC_SHA2_256_128[openssl] HMAC_SHA2_256_256[openssl]
               HMAC_SHA2_384_192[openssl] HMAC_SHA2_384_384[openssl] HMAC_SHA2_512_256[openssl]
               HMAC_SHA2_512_512[openssl] CAMELLIA_XCBC_96[xcbc]
   aead:       AES_CCM_8[aesni] AES_CCM_12[aesni] AES_CCM_16[aesni] AES_GCM_8[aesni] AES_GCM_12[aesni] AES_GCM_16[aesni]
   hasher:     HASH_SHA1[sha1] HASH_SHA2_224[sha2] HASH_SHA2_256[sha2] HASH_SHA2_384[sha2] HASH_SHA2_512[sha2]
               HASH_MD5[md5] HASH_MD4[openssl] HASH_IDENTITY[openssl]
   prf:        PRF_AES128_XCBC[aesni] PRF_AES128_CMAC[aesni] PRF_KEYED_SHA1[sha1] PRF_HMAC_MD5[openssl]
               PRF_HMAC_SHA1[openssl] PRF_HMAC_SHA2_256[openssl] PRF_HMAC_SHA2_384[openssl] PRF_HMAC_SHA2_512[openssl]
               PRF_FIPS_SHA1_160[fips-prf] PRF_CAMELLIA128_XCBC[xcbc]
   xof:        XOF_MGF1_SHA1[mgf1] XOF_MGF1_SHA224[mgf1] XOF_MGF1_SHA256[mgf1] XOF_MGF1_SHA384[mgf1]
               XOF_MGF1_SHA512[mgf1]
   dh-group:   ECP_256[openssl] ECP_384[openssl] ECP_521[openssl] ECP_224[openssl] ECP_192[openssl] ECP_256_BP[openssl]
               ECP_384_BP[openssl] ECP_512_BP[openssl] ECP_224_BP[openssl] MODP_3072[openssl] MODP_4096[openssl]
               MODP_6144[openssl] MODP_8192[openssl] MODP_2048[openssl] MODP_2048_224[openssl] MODP_2048_256[openssl]
               MODP_1536[openssl] MODP_1024[openssl] MODP_1024_160[openssl] MODP_768[openssl] MODP_CUSTOM[openssl]
               CURVE_25519[openssl] CURVE_448[openssl]
   random-gen: RNG_WEAK[openssl] RNG_STRONG[random] RNG_TRUE[random]
   nonce-gen:  [nonce]

apt install libstrongswan-standard-plugins

systemctl restart strongswan

ipsec listalgs

List of registered IKE algorithms:

encryption: AES_CBC[aesni] AES_CTR[aesni] RC2_CBC[rc2] 3DES_CBC[openssl] CAMELLIA_CBC[openssl] CAST_CBC[openssl]

BLOWFISH_CBC[openssl] DES_CBC[openssl] DES_ECB[openssl] NULL[openssl]

integrity: AES_XCBC_96[aesni] AES_CMAC_96[aesni] HMAC_MD5_96[openssl] HMAC_MD5_128[openssl] HMAC_SHA1_96[openssl]

HMAC_SHA1_128[openssl] HMAC_SHA1_160[openssl] HMAC_SHA2_256_128[openssl] HMAC_SHA2_256_256[openssl]

HMAC_SHA2_384_192[openssl] HMAC_SHA2_384_384[openssl] HMAC_SHA2_512_256[openssl]

HMAC_SHA2_512_512[openssl] CAMELLIA_XCBC_96[xcbc]

aead: AES_CCM_8[aesni] AES_CCM_12[aesni] AES_CCM_16[aesni] AES_GCM_8[aesni] AES_GCM_12[aesni] AES_GCM_16[aesni]

hasher: HASH_SHA1[sha1] HASH_SHA2_224[sha2] HASH_SHA2_256[sha2] HASH_SHA2_384[sha2] HASH_SHA2_512[sha2]

HASH_MD5[md5] HASH_MD4[openssl] HASH_IDENTITY[openssl]

prf: PRF_AES128_XCBC[aesni] PRF_AES128_CMAC[aesni] PRF_KEYED_SHA1[sha1] PRF_HMAC_MD5[openssl]

PRF_HMAC_SHA1[openssl] PRF_HMAC_SHA2_256[openssl] PRF_HMAC_SHA2_384[openssl] PRF_HMAC_SHA2_512[openssl]

PRF_FIPS_SHA1_160[fips-prf] PRF_CAMELLIA128_XCBC[xcbc]

xof: XOF_MGF1_SHA1[mgf1] XOF_MGF1_SHA224[mgf1] XOF_MGF1_SHA256[mgf1] XOF_MGF1_SHA384[mgf1]

XOF_MGF1_SHA512[mgf1]

dh-group: ECP_256[openssl] ECP_384[openssl] ECP_521[openssl] ECP_224[openssl] ECP_192[openssl] ECP_256_BP[openssl]

ECP_384_BP[openssl] ECP_512_BP[openssl] ECP_224_BP[openssl] MODP_3072[openssl] MODP_4096[openssl]

MODP_6144[openssl] MODP_8192[openssl] MODP_2048[openssl] MODP_2048_224[openssl] MODP_2048_256[openssl]

MODP_1536[openssl] MODP_1024[openssl] MODP_1024_160[openssl] MODP_768[openssl] MODP_CUSTOM[openssl]

CURVE_25519[openssl] CURVE_448[openssl]

random-gen: RNG_WEAK[openssl] RNG_STRONG[random] RNG_TRUE[random]

nonce-gen: [nonce]

And the VPN now works !

DKIM signature in Postfix on Debian

Posted on 2019/05/30 by Le_Vert

Introduction

DKIM works by interfacing a proxy in postfix to add priv/pub signature to out-going emails using a private key and opendkim.

Matching public key will be exported in the DNS server so receiving SMTP can verify in the signature in the metadata matches the public key.

It helps for non being consider as a spamer.

A friend asked me to setup this on his two servers with two different domains so we’ll be setting up the whole thing here with separate keys pair for each domain/each server and we’ll allow both servers to sign emails for these domains.

Install required packages

apt install opendkim opendkim-tools rename dnsutils

1	apt install opendkim opendkim-tools rename dnsutils

Create pub/priv keys sets

According to /usr/share/doc/opendkim/README.Debian.gz there is a tool named opendkim-genkey that can help generating the key pairs. This tool is in opendkim-tools package.

As we want to allow different servers to sign messages with different key but for the same domain, we’ll use the origin server short hostname as selector. It means the signature will be somehow prefixed with this selector, indicating the receiver SMTP which DNS entry should be queried for getting associated public key. By doing so, we’ll be able to export different public key for each server.

opendkim-genkey -b 1024 -d domain1.com -s `hostname -s` --directory=/etc/dkimkeys/
rename "s/`hostname -s`/`hostname -s`_domain1.com/" /etc/dkimkeys/`hostname -s`.{txt,private}

opendkim-genkey -b 1024 -d domain2.com -s `hostname -s` --directory=/etc/dkimkeys/
rename "s/`hostname -s`/`hostname -s`_domain2.com/" /etc/dkimkeys/`hostname -s`.{txt,private}

chown opendkim:opendkim /etc/dkimkeys/`hostname -s`_*
chmod 0600 /etc/dkimkeys/`hostname -s`_*

opendkim-genkey -b 1024 -d domain1.com -s `hostname -s` --directory=/etc/dkimkeys/

rename "s/`hostname -s`/`hostname -s`_domain1.com/" /etc/dkimkeys/`hostname -s`.{txt,private}

opendkim-genkey -b 1024 -d domain2.com -s `hostname -s` --directory=/etc/dkimkeys/

rename "s/`hostname -s`/`hostname -s`_domain2.com/" /etc/dkimkeys/`hostname -s`.{txt,private}

chown opendkim:opendkim /etc/dkimkeys/`hostname -s`_*

chmod 0600 /etc/dkimkeys/`hostname -s`_*

Now you should see the private keys as well as public key as a bind9 snippet in /etc/dkimkeys:

drwx------   2 opendkim opendkim 4.0K May 29 22:40 .
 drwxr-xr-x 109 root     root     8.0K May 29 22:11 ..
 -rw-r-----   1 root     opendkim  887 May 29 22:39 ns1234_domain1.com.private
 -rw-r-----   1 root     opendkim  329 May 29 22:39 ns1234_domain1.com.txt
 -rw-r-----   1 root     opendkim  887 May 29 22:40 ns1234_domain2.com.private
 -rw-r-----   1 root     opendkim  339 May 29 22:40 ns1234_domain2.com.txt
 -rw-r--r--   1 root     root      664 Nov  7  2015 README.PrivateKeys

drwx------ 2 opendkim opendkim 4.0K May 29 22:40 .

drwxr-xr-x 109 root root 8.0K May 29 22:11 ..

-rw-r----- 1 root opendkim 887 May 29 22:39 ns1234_domain1.com.private

-rw-r----- 1 root opendkim 329 May 29 22:39 ns1234_domain1.com.txt

-rw-r----- 1 root opendkim 887 May 29 22:40 ns1234_domain2.com.private

-rw-r----- 1 root opendkim 339 May 29 22:40 ns1234_domain2.com.txt

-rw-r--r-- 1 root root 664 Nov 7 2015 README.PrivateKeys

Register public key in DNS zones

Now you need to add the public signature in your DNS zone. In this example, the primary bind server for both domain1.com and domain2.com is running on the server itself so we can just do:

cat /etc/dkimkeys/`hostname -s`_domain1.com.txt &gt;&gt; /etc/bind/zones/domain1.com
cat /etc/dkimkeys/`hostname -s`_domain2.com.txt &gt;&gt; /etc/bind/zones/domain2.com

1 2	cat /etc/dkimkeys/`hostname -s`_domain1.com.txt >> /etc/bind/zones/domain1.com cat /etc/dkimkeys/`hostname -s`_domain2.com.txt >> /etc/bind/zones/domain2.com

This is very unlikely that method is suitable for you, but you get the idea right ?

Don’t forget to bump DNS zone serial number and reload bind

sed -i "s/20\(1\|2\)[0-9][0-9]\{2\}[0-9]\{2\}[0-9]\{2\}/`date "+%Y%m%d01"`/" /etc/bind/zones/domain1.com
sed -i "s/20\(1\|2\)[0-9][0-9]\{2\}[0-9]\{2\}[0-9]\{2\}/`date "+%Y%m%d01"`/" /etc/bind/zones/domain2.com
systemctl reload bind9

sed -i "s/20$1\|2$[0-9][0-9]\{2\}[0-9]\{2\}[0-9]\{2\}/`date "+%Y%m%d01"`/" /etc/bind/zones/domain1.com

sed -i "s/20$1\|2$[0-9][0-9]\{2\}[0-9]\{2\}[0-9]\{2\}/`date "+%Y%m%d01"`/" /etc/bind/zones/domain2.com

systemctl reload bind9

We can now check with dig that our DNS server now expose the public key:

dig -t TXT `hostname -s`._domainkey.domain1.com @127.0.0.1

1	dig -t TXT `hostname -s`._domainkey.domain1.com @127.0.0.1

Should return something like:

;; QUESTION SECTION:
;ns1234._domainkey.domain1.com. IN	TXT

;; ANSWER SECTION:
ns1234._domainkey.domain1.com. 600 IN TXT	"v=DKIM1; h=sha256; k=rsa; " "p=aaa1111bbbb3333cccc"

;; QUESTION SECTION:

;ns1234._domainkey.domain1.com. IN TXT

;; ANSWER SECTION:

ns1234._domainkey.domain1.com. 600 IN TXT "v=DKIM1; h=sha256; k=rsa; " "p=aaa1111bbbb3333cccc"

Configure OpenDKIM

Now we need to create a KeyTable file to match domain, selector and private key file. We also need a SigningTable to actually ask for signature to be added to outgoing emails.

In /etc/opendkim.conf add the following entries at bottom of files:

LogWhy                yes
KeyTable              refile:/etc/dkimkeys/KeyTable
SigningTable          refile:/etc/dkimkeys/SigningTable

LogWhy yes

KeyTable refile:/etc/dkimkeys/KeyTable

SigningTable refile:/etc/dkimkeys/SigningTable

Then we we’ll create /etc/dkimkeys/KeyTable file:

echo -e "domain1.com domain1.com:`hostname -s`:/etc/dkimkeys/`hostname -s`_domain1.com.private" &gt;&gt; /etc/dkimkeys/KeyTable
echo -e "domain2.com domain2.com:`hostname -s`:/etc/dkimkeys/`hostname -s`_domain2.com.private" &gt;&gt; /etc/dkimkeys/KeyTable

1 2	echo -e "domain1.com domain1.com:`hostname -s`:/etc/dkimkeys/`hostname -s`_domain1.com.private" >> /etc/dkimkeys/KeyTable echo -e "domain2.com domain2.com:`hostname -s`:/etc/dkimkeys/`hostname -s`_domain2.com.private" >> /etc/dkimkeys/KeyTable

The file now looks like:

domain1.com domain1.com:ns1234:/etc/dkimkeys/ns1234_domain1.com.private
domain2.com domain2.com:ns1234:/etc/dkimkeys/ns1234_domain2.com.private

1 2	domain1.com domain1.com:ns1234:/etc/dkimkeys/ns1234_domain1.com.private domain2.com domain2.com:ns1234:/etc/dkimkeys/ns1234_domain2.com.private

Now we create /etc/dkimkeys/SigningTable file:

echo "*@domain1.com domain1.com" &gt;&gt; /etc/dkimkeys/SigningTable
echo "*@domain2.com domain2.com" &gt;&gt; /etc/dkimkeys/SigningTable

1 2	echo "@domain1.com domain1.com" >> /etc/dkimkeys/SigningTable echo "@domain2.com domain2.com" >> /etc/dkimkeys/SigningTable

The file should content:

*@domain1.com domain1.com
*@domain2.com domain2.com

1 2	@domain1.com domain1.com @domain2.com domain2.com

OpenDKIM is now configured, restart it

systemctl restart opendkim

1	systemctl restart opendkim

Integrate with Postfix

On Debian systems Postfix is chrooted so there are a few additionnal steps to get it working correctly:

In /etc/opendkim.conf change the socket path to Postifx chroot:

Socket                  local:/var/spool/postfix/var/run/opendkim/opendkim.sock

1	Socket local:/var/spool/postfix/var/run/opendkim/opendkim.sock

Create proper folder in Postfix chroot and give proper permissions

mkdir -p /var/spool/postfix/var/run/opendkim
chown opendkim:opendkim /var/spool/postfix/var/run/opendkim

1 2	mkdir -p /var/spool/postfix/var/run/opendkim chown opendkim:opendkim /var/spool/postfix/var/run/opendkim

Add Postfix to opendkim group so it can write to the socket:

adduser postfix opendkim

1	adduser postfix opendkim

Enable filtering in postfix (postconf commands will edit /etc/postfix/main.cf):

Missing trailling / is not a typo !

postconf -e milter_protocol=6
postconf -e milter_default_action=accept
postconf -e smtpd_milters=unix:var/run/opendkim/opendkim.sock
postconf -e non_smtpd_milters=unix:var/run/opendkim/opendkim.sock

postconf -e milter_protocol=6

postconf -e milter_default_action=accept

postconf -e smtpd_milters=unix:var/run/opendkim/opendkim.sock

postconf -e non_smtpd_milters=unix:var/run/opendkim/opendkim.sock

Restart both services:

systemctl restart opendkim
systemctl restart postfix

1 2	systemctl restart opendkim systemctl restart postfix

Testing

You can send an email from the server itself using following commands:

echo "This is a test" | mail -s "Test DKIM domain1.com" -a "From: test@domain1.com" your@real.email
echo "This is a test" | mail -s "Test DKIM domain2.com" -a "From: test@domain2.com" your@real.email

1 2	echo "This is a test" \| mail -s "Test DKIM domain1.com" -a "From: test@domain1.com" your@real.email echo "This is a test" \| mail -s "Test DKIM domain2.com" -a "From: test@domain2.com" your@real.email

My your@real.email server runs Postfix with Amavis so I can check the header of the email I just received and I can confirm valid DKIM signature has been seen:

X-Spam-Status: No, [...] DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VERIFIED=-0.5 [...]

1	X-Spam-Status: No, [...] DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VERIFIED=-0.5 [...]

Update all DELL PowerEdge BIOS from a PXE Live Ubuntu

Posted on 2018/03/17 by Le_Vert

Quick notes from what I did today. Starting point was I was unable to update iDrac EXPRESS firmware (Uploading stuck at 95%) and these express cards don’t offer me to upgrade other BIOS anyway.

So I PXE-started the server on Ubuntu 16.05 Live CD and here’s what I had to do to be able to upgrade everything using “BIN” files (for RedHat) downloaded at DELL support website.

For the records I was upgrading the following:

R610_BIOS_C6MRW_LN_6.4.0.BIN: PowerEdgge R610
ESM_Firmware_9GJYW_LN32_2.90_A00.BIN: Embedded iDRAC controller
Network_Firmware_35RF5_LN_7.12.19.BIN: Additional 4x 1GB Broadcom card
Network_Firmware_82J79_LN_08.07.26_A00-00.BIN: Embedded Intel 1GB NICs
SAS-RAID_Firmware_9FVJ2_LN_12.10.7-0001_A13.BIN: Embedded PERC H700 RAID controller
SAS-Drive_Firmware_M2P11_LN_AS0D_A00.BIN: Seagate ST91000640SS SAS 1TB disks

Disable CDROM entryin soruces.list

sed '/^deb cdrom/d' /etc/apt/sources.list

Enable universe repository

sed -i 's/main/main universe/' /etc/apt/sources.list

Enable my HWRaid repository

echo "deb [trusted=yes] http://hwraid.le-vert.net/ubuntu xenial main" >> /etc/apt/sources.list

Install megaclisas-status to check the drive model

apt update

apt install megaclisas-status

Make sure /bin/sh is provided by bash (required by all updates)

dpkg-reconfigure dash

Select “No”

Install dependencies required by the BIN files (required by iDRAC firmware at least)
This requires to enable 32 bits support first.

dpkg --add-architecture i386
apt update

apt install libstdc++6:i386 rpm

Fake the system as being a RedHat one (required by R610 BIOS)

echo "Red Hat Enterprise Linux Server release 6.3 (Santiago)" > /etc/redhat-release

Despite tools says you’ll have to restart between each update it’s not necessary. However, double check after reboot that all have been applied by re-running the BIN files. I had to start Broadcom NIC twice, first attempt did nothing (old version still showed up after reboot)

PXE boot Debian installer with firmwares

Posted on 2018/01/12 by Le_Vert

This article is mostly a reminder for me but might be useful to others.

1. Install/prepare TFTP server

First we need a tftp server (with symlink support is better) and a network bootloader (pxelinux)

apt install tftpd syslinux-common pxelinux

Then will link the minimum files required to display a text menu for our PXE entries

ln -s /usr/lib/PXELINUX/pxelinux.0                /srv/tftp/
ln -s /usr/lib/syslinux/modules/bios/ldlinux.c32  /srv/tftp/
ln -s /usr/lib/syslinux/modules/bios/libutil.c32  /srv/tftp/
ln -s /usr/lib/syslinux/modules/bios/menu.c32     /srv/tftp/

And a default menu booting normally after a 5 seconds timeout

mkdir /srv/tftp/pxelinux.cfg

cat << 'EOF' > /srv/tftp/pxelinux.cfg/default
DEFAULT menu.c32

LABEL Normal boot
LOCALBOOT 0

PROMPT 0
TIMEOUT 50
EOF

Additional keymaps

For luxembourgish, swiss people or whoever may use different keyboard layouts, will add ability to select a keymap (in case you want to edit an existing boot entry):

We need lilo here because it includes a (broken) tool to create syslinux compatible keymaps.

apt install kbd console-data lilo

Patch the broken keytab-lilo tool using the following diff (found at https://bugzilla.syslinux.org/show_bug.cgi?id=68)

--- orig	2016-07-15 22:14:28.000000000 +0200
+++ new		2018-01-12 21:37:16.349203039 +0100
@@ -44,9 +44,9 @@
     $empty = 1;
     while () {
 	chop;
-	if (/^(static\s+)?u_short\s+(\S+)_map\[\S*\]\s+=\s+{\s*$/) {
+	if (/^(static\s+)?(u_|unsigned )short\s+(\S+)_map\[\S*\]\s+=\s+{\s*$/) {
 	    die "active at beginning of  map" if defined $current;
-	    $current = $pfx.":".$2;
+	    $current = $pfx.":".$3;
 	    next;
 	}
 	undef $current if /^};\s*$/;
EOF

Create the keymap you want (french azerty here and swiss qwertz with french variant).
For then french azerty I found a few additional remaping here: https://forums.archlinux.fr/viewtopic.php?t=11953

mkdir /srv/tftp/keymaps
keytab-lilo -p 60=46 -p 92=60 -p 124=62 \
            /usr/share/keymaps/i386/qwerty/us.kmap.gz \
            /usr/share/keymaps/i386/azerty/fr-latin1.kmap.gz \
            > /srv/tftp/keymaps/fr.ktl
keytab-lilo /usr/share/keymaps/i386/qwerty/us.kmap.gz \
            /usr/share/keymaps/i386/qwertz/fr_CH-latin1.kmap.gz \
            > /srv/tftp/keymaps/fr-ch.ktl

Uninstall lilo:

apt purge lilo

And add entry in the menu to select an alternative keymap

ln -s /usr/lib/syslinux/modules/bios/kbdmap.c32  /srv/tftp/
ln -s /usr/lib/syslinux/modules/bios/libcom32.c32  /srv/tftp/
cat << 'EOF' >> /srv/tftp/pxelinux.cfg/default

LABEL Switch to french AZERTY
KERNEL kbdmap.c32
APPEND keymaps/fr.ktl

LABEL Switch to swiss french QWERTZ
KERNEL kbdmap.c32
APPEND keymaps/fr-ch.ktl
EOF

Debian installer with non-free firmware

Extract kernel and initrd from the Debian netboot archive:

mkdir /srv/tftp/debian-stretch-amd64-netinstall
wget -q -O- http://ftp.fr.debian.org/debian/dists/stretch/main/installer-amd64/current/images/netboot/netboot.tar.gz \
  | tar xvz -C /srv/tftp/debian-stretch-amd64-netinstall/ \
            ./debian-installer/amd64/initrd.gz \
            ./debian-installer/amd64/linux \
            --strip-components=3

Download firmware file and append to original initrd:

wget -q -O /srv/tftp/debian-stretch-amd64-netinstall/firmware.cpio.gz \
  http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/stretch/current/firmware.cpio.gz

cat /srv/tftp/debian-stretch-amd64-netinstall/initrd.gz \
    /srv/tftp/debian-stretch-amd64-netinstall/firmware.cpio.gz \
    > /srv/tftp/debian-stretch-amd64-netinstall/initrd_firmware.gz

And add it to the boot menu:

cat << 'EOF' >> /srv/tftp/pxelinux.cfg/default

LABEL Debian Stretch NetInstall (firmware) amd64
KERNEL debian-stretch-amd64-netinstall/linux
APPEND initrd=debian-stretch-amd64-netinstall/initrd_firmware.gz
EOF

Mimic IPv4 docker behavior for IPv6 with Shorewall and NAT

Posted on 2017/09/29 by Le_Vert

Hi,

Docker IPv6 support is messed up. Instead of sharing a non-routed IPv6 prefix between host and containers, just like it does for IPv4, docker team decided to implement a full IPv6 support, with routing and everything.
The point is: most of people just want basic IPv6 inside the container (in my case I’m doing monitoring checks) and would love to have it working out of the box. Honestly, who wants a dedicated routed prefix for its IPv6 containers ? Certainly not me.

So here’s the trick: we’re going to use a chosen prefix between docker daemon and its host and we’ll use shorewall6 to NAT it when going outside. Basically it’ll mimic IPv4 behavior.

So first, make sure IPv6 is working correctly on host (actually while writing this doc, it was not on my sample setup :D):

root@server:~# ping6 -n google.lu
PING google.lu(2a00:1450:4001:816::2003) 56 data bytes
64 bytes from 2a00:1450:4001:816::2003: icmp_seq=1 ttl=57 time=10.3 ms
64 bytes from 2a00:1450:4001:816::2003: icmp_seq=2 ttl=57 time=10.3 ms

root@server:~# ping6 -n google.lu

PING google.lu(2a00:1450:4001:816::2003) 56 data bytes

64 bytes from 2a00:1450:4001:816::2003: icmp_seq=1 ttl=57 time=10.3 ms

64 bytes from 2a00:1450:4001:816::2003: icmp_seq=2 ttl=57 time=10.3 ms

Then, define a IPv6 prefix to be use inside docker by editing /etc/docker/daemon.json

{
	"ipv6": true,
	"fixed-cidr-v6": "2a00:1450::/64"
}

{

"ipv6": true,

"fixed-cidr-v6": "2a00:1450::/64"

}

Create the file if not existing yet.

Restart docker daemon:

systemctl restart docker

1	systemctl restart docker

Now your container should have an IP range in the range (and a default gateway):

root@server:~# docker <container_name> -ti centreon ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
228: eth0@if229: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP 
    inet6 2a00:1450::242:ac11:2/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

root@server:~# docker <container_name> -ti centreon ip -6 a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

228: eth0@if229: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP

inet6 2a00:1450::242:ac11:2/64 scope global nodad

valid_lft forever preferred_lft forever

inet6 fe80::42:acff:fe11:2/64 scope link

valid_lft forever preferred_lft forever

root@server:~# docker exec -ti <container_name> ip -6 r
2a00:1450::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
default via 2a00:1450::1 dev eth0  metric 1024

root@server:~# docker exec -ti <container_name> ip -6 r

2a00:1450::/64 dev eth0 proto kernel metric 256

fe80::/64 dev eth0 proto kernel metric 256

default via 2a00:1450::1 dev eth0 metric 1024

You host should be able to reach the container too:

root@server:~# ping6 2a00:1450::242:ac11:2
PING 2a00:1450::242:ac11:2(2a00:1450::242:ac11:2) 56 data bytes
64 bytes from 2a00:1450::242:ac11:2: icmp_seq=1 ttl=64 time=0.076 ms
64 bytes from 2a00:1450::242:ac11:2: icmp_seq=2 ttl=64 time=0.099 ms

root@server:~# ping6 2a00:1450::242:ac11:2

PING 2a00:1450::242:ac11:2(2a00:1450::242:ac11:2) 56 data bytes

64 bytes from 2a00:1450::242:ac11:2: icmp_seq=1 ttl=64 time=0.076 ms

64 bytes from 2a00:1450::242:ac11:2: icmp_seq=2 ttl=64 time=0.099 ms

Here we go… Now all we need to do is make a source NAT of 2a00:1450::/64 when going out of eth0 on the host server.

For this, we’ll use shorewall:

apt -y install shorewall6

1	apt -y install shorewall6

And we’ll create a default configuration based on provided examples:

cp /usr/share/doc/shorewall6/examples/one-interface/{policy,interfaces,rules,zones} /etc/shorewall6/

1	cp /usr/share/doc/shorewall6/examples/one-interface/{policy,interfaces,rules,zones} /etc/shorewall6/

Default configuration can work but I’ll tweak it a bit.

In /etc/shorewall6/interfaces I renamed the interface to loc instead of net and disable routeur advertisment (IPv6 gateway statically defined in /etc/network/interfaces) and I need to define the docker bridge interface:

loc     eth0            tcpflags,accept_ra=0,sourceroute=0
dockr   docker0         tcpflags,accept_ra=0,sourceroute=0

1 2	loc eth0 tcpflags,accept_ra=0,sourceroute=0 dockr docker0 tcpflags,accept_ra=0,sourceroute=0

In /etc/shorewall6/zones there must be one ipv6 zone for each interface:

fw      firewall
loc     ipv6
dockr   ipv6

fw firewall

loc ipv6

dockr ipv6

In /etc/shorewall6/policy I give full access to host server to any network, docker has access to local network (but not to its host) and I block everything else

fw              all             ACCEPT
dockr           loc             ACCEPT
all             all             DROP            info

fw all ACCEPT

dockr loc ACCEPT

all all DROP info

Then, I create a few overrides in /etc/shorewall6/rules to authorize Ping and SSH from local network:

# From docker network Ping(ACCEPT) dockr fw MySQL(ACCEPT) dockr fw SNMP(ACCEPT) dockr fw ACCEPT dockr fw tcp 5666 # From local netowork Ping(ACCEPT) loc fw SSH(ACCEPT) loc fw SNMP(ACCEPT) loc fw ACCEPT loc fw tcp 5666 HTTP(ACCEPT) loc fw

1
2
3
4
5
6
7
8
9
10
11
12

# From docker network
Ping(ACCEPT)    dockr           fw
MySQL(ACCEPT)   dockr           fw
SNMP(ACCEPT)    dockr           fw
ACCEPT          dockr           fw              tcp     5666

# From local netowork
Ping(ACCEPT)    loc             fw
SSH(ACCEPT)     loc             fw
SNMP(ACCEPT)    loc             fw
ACCEPT          loc             fw              tcp     5666
HTTP(ACCEPT)    loc             fw

As you can see, there’s a few more rules: at least one containers uses a MariaDB server on the host, so I permit that. Also I permit Nagios NRPE and SNMP from local network to host (fw) for monitoring purpose and I also accept the same from container to host because there’s a monitoring container that will actually monitor its own host.

Now we need to NAT the private IPv6 subnet used by docker in /etc/shorewall6/snat

MASQUERADE		2a00:1450::/64		eth0

1	MASQUERADE 2a00:1450::/64 eth0

Finally we’ll enable IP_FORWARDING in */etc/shorewall6/shorewall.conf* by setting it to yes and restart shorewall:

sed -i 's!IP_FORWARDING=.*!IP_FORWARDING=Yes!' /etc/shorewall6/shorewall6.conf
systemctl enable shorewall6
systemctl start shorewall6

sed -i 's!IP_FORWARDING=.*!IP_FORWARDING=Yes!' /etc/shorewall6/shorewall6.conf

systemctl enable shorewall6

systemctl start shorewall6

And that’s it:

root@server:~# docker exec -ti <container_name> ping6 -n google.lu
PING google.lu(2a00:1450:4001:81b::2003) 56 data bytes
64 bytes from 2a00:1450:4001:81b::2003: icmp_seq=1 ttl=56 time=10.6 ms
64 bytes from 2a00:1450:4001:81b::2003: icmp_seq=2 ttl=56 time=10.5 ms

root@server:~# docker exec -ti <container_name> ping6 -n google.lu

PING google.lu(2a00:1450:4001:81b::2003) 56 data bytes

64 bytes from 2a00:1450:4001:81b::2003: icmp_seq=1 ttl=56 time=10.6 ms

64 bytes from 2a00:1450:4001:81b::2003: icmp_seq=2 ttl=56 time=10.5 ms

Fixing VMware vCenter template customization for Debian Stretch (nic detected as “ether”)

Posted on 2017/07/27 by Le_Vert

Hello,

I’m a big fan of Foreman, I use it everywhere to spawn my virtual machines (mostly with VMWare vCenter or AWS) and then apply directly Puppet classes on it to get a fully configured new host in a few clicks. Maybe I’ll write about it one day, let’s see.

Anyway, this week theme was mostly “Let’s upgrade from Jessie to Stretch, I’m craving Python 3.5 and the new async/await syntax)”.
Sadly, it went wrong. I was unable to use my Foreman anymore against ESX 6.0 because when injecting the customization XML file (used to define IP settings within the VM through open-vm-tools) the resulting VM had no network set.
After looking at what happened, I figured out /etc/network/interfaces had been created wrong: instead of using eth0 (yes, I disabled predicitve interface name in my template) it was all set like the interface was named ether. Uh ?

Quick Google search with “debian stretch vmware ether” lead me to the following GitHub bug opened against open-vm-tools. Sadly the issue wouldn’t come from open-vm-tools: this issue comes from a VMWare script not parsing correctly current ifconfig output (yeah, I added net-tools in my template too).

Here is an extract of the net-tools package NEWS.Debian file:

net-tools (1.60+git20161116.90da8a0-1) unstable; urgency=medium

  After 15 years without upstream development, net-tools is being worked on
  again, fixing many long-standing issues.

  The bad news is that the output of many commands has changed, and it is sure
  to break scripts that relied on parsing it.

  If you have customs scripts that use any of these commands, please make sure
  they still work after this upgrade:

    netstat, ifconfig, ipmaddr, iptunnel, mii-tool, nameif, plipconfig, rarp,
    route, slattach, arp.

  Apologies in advance for the trouble that this may cause, but maintaining a
  separate version of net-tools just to keep the old format is something I am
  not able to do.

net-tools (1.60+git20161116.90da8a0-1) unstable; urgency=medium

After 15 years without upstream development, net-tools is being worked on

again, fixing many long-standing issues.

The bad news is that the output of many commands has changed, and it is sure

to break scripts that relied on parsing it.

If you have customs scripts that use any of these commands, please make sure

they still work after this upgrade:

netstat, ifconfig, ipmaddr, iptunnel, mii-tool, nameif, plipconfig, rarp,

route, slattach, arp.

Apologies in advance for the trouble that this may cause, but maintaining a

separate version of net-tools just to keep the old format is something I am

not able to do.

Wow, that’s a pretty dangerous move you did here….

The script creating the network configuration is actually a piece of Perl crap copied directly from the vCenter server into the VM filesystem. Yeah, that sounds like black magic but the good news is that’s it’s Perl, so it’s fixable.

So I searched for this “Customization.pm” file on my vCenter Windows server and I found it here:
C:\Program Files\VMware\vCenter Server\vpxd\imgcust\linux\imgcust-scripts\Customization.pm

I managed quite easily to understand what was wrong, and I must say that original output parsing was pretty cheap.
Anyway, here’s a better one that just works:

+++ Customization.pm.new	2017-07-27 20:21:35.559476473 +0200
@@ -756,17 +756,17 @@
    if (! defined $ifcfgResult) {
       $ifcfgResult = Utils::ExecuteCommand('/sbin/ifconfig -a');
    }
+  
+   my @interfaces = split/\n\n/, $ifcfgResult;
 
-   my $result = undef;
-
-   my $macAddressValid = ($macAddress =~ /^([0-9a-f]{2}:){5}[0-9a-f]{2}$/i);
-
-   if ($macAddressValid &amp;&amp;
-      ($ifcfgResult =~ /^\s*(\w+?)(:\w*)?\s+.*?$macAddress/mi)) {
-      $result = $1;
+   foreach my $interface (@interfaces) {
+      if ($interface =~ /$macAddress/) {
+	      print($interface);
+          if ($interface =~ m/^(\w+)(:|\s)/) {
+              return $1;
+          }
+      }
    }
-
-   return $result;
 }
 
 sub GetInterfaceByMacAddressIPAddrShow

+++ Customization.pm.new 2017-07-27 20:21:35.559476473 +0200

@@ -756,17 +756,17 @@

if (! defined $ifcfgResult) {

$ifcfgResult = Utils::ExecuteCommand('/sbin/ifconfig -a');

}

+ my @interfaces = split/\n\n/, $ifcfgResult;

- my $result = undef;

- my $macAddressValid = ($macAddress =~ /^([0-9a-f]{2}:){5}[0-9a-f]{2}$/i);

- if ($macAddressValid &&

- ($ifcfgResult =~ /^\s*(\w+?)(:\w*)?\s+.*?$macAddress/mi)) {

- $result = $1;

+ foreach my $interface (@interfaces) {

+ if ($interface =~ /$macAddress/) {

+ print($interface);

+ if ($interface =~ m/^(\w+)(:|\s)/) {

+ return $1;

+ }

}

- return $result;

}

sub GetInterfaceByMacAddressIPAddrShow

Nothing to restart, this file in copied everytime you apply customization to a template. You’ll find attached a text version of the patch: vcenter_Customization_pm.diff

Good luck!

Policy Based Routing (PBR) with Shorewall to migrate a server

Posted on 2017/07/24 by Le_Vert

Hey,

Today I’m doing pure sysadmin work and I’ve been asked to migrate several servers from an obsolete IP range 192.168.x to 10.x. Things were quite easy until I reached the internal mail server that can be used by hundreds field hardware as a relay server. Everybody is supposed to use DNS entry but I won’t trust that.

So my idea is to switch eth0 to the new network and keep a new eth1 in the old one to keep the service working and be able to log what’s using the obsolete address.
There’s just a little problem: if my default gateway is on eth0, any packet entering eth1 from a routed network (would work for what’s connected in the legacy local network) will be answered using the default gateway on eth0. That’s asymmetric routing and that just doesn’t work.

Okay, so how do I solve that ? With Shorewall of course ! The idea if to tag any packet entering eth1 with a different mark than the ones coming throught eth0 and provide different routing table for each mark. I’ll do this on CentOS today but it should be basically the same for any Linux system. Shorewall is usually available everywhere but you can try doing this by hand with “ip” and “iptables”. Looks like a lot of pain to me, though.
Having both address routed and working is a nice step but it’s pretty useless if I have no way to find out who’s using the obsolete address so we’ll use Shorewall to log these access and create a specific rsyslog/logrotate configuration to get a dedicated log.

First, we’ll change the network configuration to have both interfaces up with a default gateway only on the first interface (connected to the new network). The gateway will be later overridden by Shorewall but it’s always saner to have a default configuration working, even with limited feature.

So make sure to create proper ifcfg-eth0 and ifcfg-eth1 in /etc/sysconfig/network-scripts and make sure to have only GATEWAY defined on the new network. You should also make sure that the server is reachable on its new address and reachable on the old address with a machine directly connected to the legacy network.

Let’s continue with a very basic Shorewall configuration. yum -y install shorewall and then make sure to have the three following files in /etc/shorewall:

interfaces – List of network adapter handled by Shorewall
policy – Default firewall policies between each zone
providers – This one is PBR specific, we’ll use this to mark packets
rules – Overrides default policies with port/host rules
shorewall.conf – Global settings
zones – Map interfaces to firewall zones

If you miss one, copy it from /usr/share/shorewall/configfiles/

So let’s do a few adjustments in shorewall.conf first:
IP_FORWARDING=No (No this machine SHOULD never be used a gateway between legacy and new network, we're not here to create security flows ;-)) DISABLE_IPV6=Yes (Sadly, there's no IPv6 here so it's better to let Shorewall4 kill the whole stack) LOGTAGONLY=Yes (Change the way Shorewall generate log prefix, otherwise ours will be too long and get shortened)

Now defines the interfaces in rules:

loc   eth0   tcpflags,logmartians,nosmurfs,sourceroute=0
old   eth1   tcpflags,logmartians,nosmurfs,sourceroute=0

1 2	loc eth0 tcpflags,logmartians,nosmurfs,sourceroute=0 old eth1 tcpflags,logmartians,nosmurfs,sourceroute=0

And map them to IPv4 zones:

fw      firewall
loc     ipv4
old     ipv4

fw firewall

loc ipv4

old ipv4

fw is a default zone meaning “myself”.

And we create a default policy allowing the machine itself to reach legacy and new network zones and blocking any incoming packets.

fw              loc,old         ACCEPT
all             all             DROP            info

1 2	fw loc,old ACCEPT all all DROP info

Finally we’ll add a set of default rules to be at least, to SSH the server again

Ping(ACCEPT)    loc         fw
SSH(ACCEPT)     loc         fw

1 2	Ping(ACCEPT) loc fw SSH(ACCEPT) loc fw

Just like in policy file, you can use loc,old if you want to permit ping and SSH from the old network too.

I’ll also add a few rules to permit mail related services from the new zone:

SMTP(ACCEPT)            loc     fw
Submission(ACCEPT)      loc     fw
IMAP(ACCEPT)            loc     fw

SMTP(ACCEPT) loc fw

Submission(ACCEPT) loc fw

IMAP(ACCEPT) loc fw

Okay, now we can enable and start Shorewall.

systemctl enable shorewall
systemctl start shorewall

1 2	systemctl enable shorewall systemctl start shorewall

Now we’ll ask Shorewall to mark packets differently according to the incoming interface. This will be done in providers file.

NEW  1  1  -  eth0  10.0.0.254     track
OLD  2  2  -  eth1  192.168.0.254  track

1 2	NEW 1 1 - eth0 10.0.0.254 track OLD 2 2 - eth1 192.168.0.254 track

Last column is the gateway to use on each network.

Let’s permit mail-related traffic from the legacy network but ask Shorewall to log these packets. Add the following in rules file:

SMTP(ACCEPT):info:MailMigration        old  fw
Submission(ACCEPT):info:MailMigration  old  fw
IMAP(ACCEPT):info:MailMigration        old  fw

SMTP(ACCEPT):info:MailMigration old fw

Submission(ACCEPT):info:MailMigration old fw

IMAP(ACCEPT):info:MailMigration old fw

Reload Shorewall and try to telnet tcp/25 from a routed network, both IPs are now working !

systemctl reload shorewall

1	systemctl reload shorewall

If you check /var/log/messages you will see log like:
Jul 24 16:55:39 mailsrv kernel: Shorewall:MailMigration:ACCE IN=eth1 OUT= MAC=XXX SRC=192.168.55.4 DST=192.168.0.10 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=22405 DF PROTO=TCP SPT=39474 DPT=25 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0x2

You can also check your routing tables with ip route show table.
ip route show table main shows no more default gateway
ip route show table 1 shows local route for eth0 network and default gateway of the new network. It’ll be used for packet tagged as 1.
ip route show table 2 shows local route for eth1 network and default gateway of the legacy network. It’ll be used for packet tagged as 2 (Note the log above with MARK=0x2).

Your server is now completely accessible from both networks and you can easily monitor the log file to find clients still using the legacy address. But we can make it a lot easier by asking rsyslog to create a separate log file with these specific messages:

Create /etc/rsyslog.d/mailmigration.conf with the following content:

$FileOwner root
$FileGroup root
$Umask 0000
$FileCreateMode 0644
:msg, startswith, "Shorewall:MailMigration:" /var/log/mailmigration.log
& stop

$FileOwner root

$FileGroup root

$Umask 0000

$FileCreateMode 0644

:msg, startswith, "Shorewall:MailMigration:" /var/log/mailmigration.log

& stop

And the associated logrotate /etc/logrotate.d/mailmigration file to avoid having a single never ending file:

/var/log/mailmigration.log
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

/var/log/mailmigration.log

{

missingok

sharedscripts

postrotate

/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

endscript

}

If you want to go further for a more automated way of handling this, I’d definitely suggest having a look at Rsyslog AMQP module to publish event to a RabbitMQ and write a quick Python consumer to parse and notify “Someone” (may I suggest calling some API to create an internal support ticket ?) using Pika. The “worker.py” file should be enough for testing, just try/except/ch.basic_nack your handler so the message goes back in queue in case of failure.

Building native Python 3.4 module with Pip on Windows

Posted on 2017/03/27 by Le_Vert

Hello,

Yesterday I decided to use Python setproctitle module in a project to rename Python script process name (for pretty displaying in nestat, ps…).
RPM package for CentOS 7 was done very quickly by modifying current dev Fedora one to get Python34 flavor for my old CentOS but I try to keep compatibility of my code for Windows too (mostly for development purpose of colleagues).

As usual, I would just go for “pip install setproctitle” on Windows (I would clearly advice to NEVER do that on production but it’s fine for developing).
Sadly it failed with the following error:
error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat).

According to Google this error is quite famous but most of the people seem to be trying to fix it without having any clue of what is really going on.
The root of this issue is that renaming a process in something really specific and thus platform dependent. If you look at setproctitle code you’ll see it’s all C code with specific section for each family of operating system. So we are having to issue installing this module on Windows because:

You need a compiler, but unlike on Linux you need the same compiler that the one Python team used when building the Python interpreter you have installed
You will probably also need Windows SDK, because setproctitle is very likely to use Windows low-level headers

According to pip error message when installing setproctitle module, I need Visual Studio 10.0 compiler. Okay.
Thanks to Wikipedia, I’m now aware that version 10.0 is actually Visual Studio 2010.
Microsoft confirms this but adds an interesting information: Visual Studio 2010 is a commercial software, so I need a free alternative which is Microsoft Windows SDK for Windows 7 and .NET Framework 4 embedding Visual Studio 2010 compiler.

I’d suggest getting the ISO version instead because the previous link in an online installer. It may not work anymore next time you need to install it…

A funny thing: you’ll be prompted for three different ISO file without any information about what the difference is… So here is the explanation:

GRMSDK_EN_DVD.iso: This is the regular X86 Windows running in 32 bits mode
GRMSDKIAI_EN_DVD.iso: Intel Itanium 64 bits, you don’t want that
GRMSDKX_EN_DVD.iso: X64 version, that’s probably the one you need

If you get the wrong one, the installer will fail with weird error message saying there’s an MSI file missing!

Before trying to install this, uninstall any “Visual Studio 2010” related software, especially the classic Microsoft Visual Studio 2010 Redistribuable X86 and X64 which are very likely to be installed already. Otherwise, the SDK will fail to install without any understandable error message (but you’re free to give it a try and try to figure out what’s going on in the Windows Installer log file, good luck)

I also had trouble running the installer from a network mapped drive, so you can safely extract ISO content with 7-Zip but you might have to copy the folder locally before running it (any feedback would be appreciated in comments if you give a try).

You may now think you’re ready but wait… There’s more.
It seems Windows SDK package installs a broken Visual Studio distribution: KB2519277 (FIX: Visual C++ compilers are removed when you upgrade Visual Studio 2010 […] if Windows SDK v7.1 is installed). According to the title, it’s not exactly what we are doing but you really need that, get VC-Compiler-KB2519277.exe here:

Last but no least, despite Microsoft released a fix to repair a broken Visual Studio installation from the SDK package, they still managed to release it broken: it’s not working on X64, there’s a missing BAT file to set environment variables when running from an X64 shell 😀 No kidding…

Even worse, it has been reported to Microsoft but they closed the issue with no explanation: https://connect.microsoft.com/VisualStudio/feedback/details/510784/vcvarsall-bat-amd64-environment-is-missing

Hopefully some people at StackOverflow fixed the issue by themselves.

I made a batch script so anyone can just run the script and enjoy the fix:

@ECHO OFF

NET SESSION >nul 2>&1
IF %ERRORLEVEL% EQU 0 (
    ECHo ADMINISTRATOR PRIVILEGES AVAILABLE
) ELSE (
    ECHO PLEASE RUN BY DOING RIGHT CLICK
    ECHO RUN AS ADMINISTRATOR
    PAUSE
    EXIT
)

MD "%PROGRAMFILES(x86)%\Microsoft Visual Studio 10.0\VC\bin\amd64\"

ECHO CALL "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 1> "%PROGRAMFILES(x86)%\Microsoft Visual Studio 10.0\VC\bin\amd64\vcvars64.bat"

PAUSE

@ECHO OFF

NET SESSION >nul 2>&1

IF %ERRORLEVEL% EQU 0 (

ECHo ADMINISTRATOR PRIVILEGES AVAILABLE

) ELSE (

ECHO PLEASE RUN BY DOING RIGHT CLICK

ECHO RUN AS ADMINISTRATOR

PAUSE

EXIT

)

MD "%PROGRAMFILES(x86)%\Microsoft Visual Studio 10.0\VC\bin\amd64\"

ECHO CALL "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 1> "%PROGRAMFILES(x86)%\Microsoft Visual Studio 10.0\VC\bin\amd64\vcvars64.bat"

PAUSE

Again, it cannot be run from a network drive (wtf, Windows really…) so you’ll have to create this script on your desktop with “.bat” extension and run it with administrator privileges using right click.

Now you can go back to pip and enjoy the package building and installing successfully 🙂

PS: Did I mention setproctitle 1.1.10 module is real good shit ? If you’re running tones of Python processes, especially related to networking you may benefit from a renamed process when using ps or netstat !

A working Microsoft RDP (remote desktop) client

Posted on 2017/02/25 by Le_Vert

Hey,

Recent Windows Server release (like 2012) seems to require some additional feature the good old “rdesktop” tools do not handle. Here is what happens when connecting:

Autoselected keyboard map en-us
ERROR: CredSSP: Initialize failed, do you have correct kerberos tgt initialized ?
Failed to connect, CredSSP required by server.

Many people around Internet suggest disabling something on the server but it means disabling some security feature. Moreover, you might need to use RDP to disable this (ZeroDivisionError) and you may not be allowed to do so. Anyway, shitty answser.

Here is the proper one: https://github.com/FreeRDP/FreeRDP

This client just works but has the same issue as rdesktop: it’s highly stupid. For instance, look at the error message above and notice “Autoselected keyboard map en-us”.
Sorry, what ? It’s not because I’m using en_US locale that I’m actually staying in the United States and using a regular ANSI QWERTY keyboard. In fact, I’m not, not at all.
Another issue is the screen size setting which seems to be always set to 1024×768 which is a pitty nowadays, everybody uses at least “FullHD” screen.

So I made a shell wrapper script implementing dynamic screen size selection to 90% of your current display (configurable) and setting the right keymap according to your keyboard layout and variant (layout=ch, variant=fr for me, which is a french oriented QWERTZU layout used in Luxembourg and called by Windows “Swiss French”).

It also feature a configuration file to override defaults and some handy default options to share clipboard and home disk with the remote target. All you have to do is to put saner-xfreerdp in /usr/local/bin/ and use it instead of the real binary.

Get the script here: https://github.com/eLvErDe/saner-xfreerdp

Here is an very simple usage example:

user@host:~$ saner-xfreerdp -u username -a some-srv-01.domain.lan
INFO: Detected active screen on monitor DVI-0 with width=1920 and height=1200
INFO: Will use resized resolution of 1728×1080

INFO: Running xfreerdp +clipboard +home-drive /u:”username” /v:”some-srv-01.domain.lan” /kbd:”Swiss French” /w:1728 /h:1080

[xrdp logs…]
Password:

Debugging “no output” in Nagios or Centreon

Posted on 2017/02/02 by Le_Vert

You just set up a new test that was running perfectly fine when ran by hand but fails completely after integration in the monitoring software ?
Of course, you suspect that the actually run command in invalid, thanks to parameters, quotes, escapes or whatever but you’re having hard time to figure out what was run exactly…

Been there, done that. But here’s a magic trick:
Let’s do some kind of “ps | tail -f | grep” on the monitoring poller itself:

while true; do ps aux | grep check_script_name | grep -v grep; done

Now, trigger a forced check and get the full command on your terminal.
Some quotes might be missing because ps aux doesn’t show the argument separator but I guess that could be workarounded with a real script querying /proc/${pid}/cmdline that contains \0 arg separator…

blog.le-vert.net

No bullshit, only Linux stuff

StrongSwan: Encryption not supported

Introduction

Debian tricked me

Investigations and resolution

DKIM signature in Postfix on Debian

Introduction

Install required packages

Create pub/priv keys sets

Register public key in DNS zones

This is very unlikely that method is suitable for you, but you get the idea right ?

Configure OpenDKIM

Integrate with Postfix

Testing

Update all DELL PowerEdge BIOS from a PXE Live Ubuntu

PXE boot Debian installer with firmwares

1. Install/prepare TFTP server

Additional keymaps

Debian installer with non-free firmware

Mimic IPv4 docker behavior for IPv6 with Shorewall and NAT

Fixing VMware vCenter template customization for Debian Stretch (nic detected as “ether”)

Policy Based Routing (PBR) with Shorewall to migrate a server

Building native Python 3.4 module with Pip on Windows

A working Microsoft RDP (remote desktop) client

Debugging “no output” in Nagios or Centreon