Tag Archives: security

Building a l2tp/IPsec VPN based around a OpenBSD head-end – Part 1

This is the first in a series of posts to cover building a l2tp/IPsec VPN service which remote users (road warriors) connect to.
In this post I will begin with getting OpenBSD setup as the head-end & follow up with subsequent posts to cover configuration of various platforms as clients which compose the road warriors.
Undeadly featured an article on configuring OpenBSD in 2012, things have improved since this article was posted and some of the steps are no longer required, hence I will go over the process again here.

It’s assumed you have an install of OpenBSD running that’s setup as a gateway and communicating on the network, we will continue from there.

The following snippet of config needs to be added to your PF config (/etc/pf.conf by default). It unconditionally permits the IPsec ESP & AH protocols intended for the OpenBSD host, as well as any UDP traffic for ISAKMP and to support NAT traversal.
pass quick proto { esp, ah } from any to self
pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state
pass on enc0 from any to self keep state (if-bound)

A minimal PF config which just permits the establishment of a VPN tunnel might look like the following

set skip lo
block return
pass quick proto { esp, ah } from any to self
pass quick proto udp from any to self port {isakmp, ipsec-nat-t} keep state
pass on enc0 from any to self keep state (if-bound)

By only permitting isakmp, it enforces having a working IPsec config before anything else happens whereas permitting UDP port 1701 would permit the establishment of a l2tp tunnel without IPsec which in this scenario would likely be undesired.

A basic IPsec config to use a pre-shared key.The default ciphers used for main & quick mode are documented in ipsec.conf(5). The IP address 1.2.3.4 is configured on the OpenBSD host which connections will be accepted on.

ike passive esp transport proto udp from 1.2.3.4 to any port 1701 psk "password"

Note, the OpenBSD defaults are too high for establishing a connection using the networking preferences on Apple devices and so would need to be restricted down to auth "hmac-sha1" enc "3des" group modp1024 which is not recommended, configuring Apple systems will be covered as a separate article.

The default npppd config (/etc/npppd/nppd.conf) works as-is, without any further changes required. That is unless you prefer to use RADIUS for accounting, instead of local user accounts.

myuser:\
    :password=mypass:\
    :framed-ip-address=10.0.0.111:

npppd is set to use pppx(4) interfaces for established sessions, in order for these interfaces to work correctly, pipex(4) needs to be enabled.

sysctl net.pipex.enable=1

and adding net.pipex.enable=1 to /etc/sysctl.conf so it’s set on boot.

Note, hosts missing this commit (5.8-RELEASE and snapshots from today & prior) will suffer a panic on the OpenBSD host upon establishment of a session by clients, if pipex(4) is not enabled.

Start isakmpd & npppd with

isakmpd -K
npppd

Load your ipsec.conf with
ipsecctl -f /etc/ipsec.conf

Your host should be ready to accept VPN connections, set this services to be started on boot by adding the following to /etc/rc.conf.local
isakmpd_flags="-K"
ipsec=YES
npppd_flags=""

Adventures in Open Source Software: Dealing with Security

I had the opportunity to give this talk at the London chapter of DefCon, DC4420 and censecutivily at London Perl Mongers technical meeting last week.
The subject of the talk was all the factors outside of doing security resceach which can make the process of dealing with advisories a daunting or a seamless process. As observed during the last year, while working as part of a security team.

Slides

Intro:
pkgsrc is a crossplatform packaging system by the NetBSD project, forked from the FreeBSD ports in the late 90’s, initially the primary target was NetBSD but with the portability focus of the project, the list of supported flatforms has grown to a list of 23 operating systems (16 out of those 23 are currently actively worked on). Within the pkgsrc project, there is a dedicated security team whose responsibility is to audit published vulnerabilities and ensure that those which apply to software we offer packages for are listed in a file. Users download this file & use it to check their installed packages.
Other open source projects have teams who are more involved and participate in the security research process and publish their own advisories, such as Debian or Redhat but that is not  main focus of our team.
There may be several reasons for this, during my talk I refered to the aquisition of a security company by Redhat, but looking up Redhat on Wikipedia, there doesn’t appear to be anything to suggest that.
I can say that for the pkgsrc-security team, the role is focused on filtering information and ensuring that items are listed in the pkg-vulnerabilities file, maintainers are notified (if there is one) and co-ordinating with the release engineering team so that necessary commits are pulled into the relevant branches. This is because we try to avoid dealing with development within our tree and opt to co-ordinate with upstream to submit fixes. Majority of our changes focus on removing assumptions to ensure things are built in a consistent manner and allow the software to be packaged how we like.

Dealing with advisories:
The advisories which we receive range in quality / detail.
A personal favourite are the drupal advisories. We offer the drupal core as a package but not any of the 3rd party modules. Their advisories clearly indicate if they apply to core or any published 3rd party modules and which scenario is needed for the vulnerability to be exploitable eg the user must be able to upload content.

The opposite of that is independently published advisories without any co-ordination with affected parties or independently published advisories which are disputed or not acknowledge from upstream. In these situation the role becomes more involved in order to work out if there is clearly an issue or not.

Then there’s Oracle advisories, we can confirm there’s one or more problems in the following versions of software, no more details than that. Upgrade to this version at a minimum to fix said issue(s). Here’s a chart so you can evaluate the risk.

It can be that upstream has actually made an announcement with the details of an issue in public but the mitre website will still lists the CVE as reserved. Ideally you’d like to list the mitre site in pkg-vulnerabilities as it’s where IDs are assigned and it’s self referencing (url will contain the CVE id). But it’s a terrible thing to do to a user. “You have a package installed which is vulnerable to the following type of issue follow this link to not find out any more information about it. Go fish”. Or maybe you have no choice.

Project Websites:
If the published advisory come via a Linux distribution it can be common that the fix references a binary package for users to install or perhaps further information required. In the 2000’s Soureceforge was a popular host for open source projects usually complimented with a separate web page of some kind, it’s now common to have projects which solely exists as an authoritative repo on github. In either scenario, a dedicated section for publishing security information is usually not found. This trend is also prevelant in large commercially backed projects, which play an extremely critical role. Projcts such as ICU (International Components for Unicode), a project by IBM which deals with unicode, an issue in ICU can mean an issue in chrome/chromium, java.

There are also projects like Qemu which have a security page for submitting vulnerability information but never publish advisories themselves. It is common for advisories to reference a git commit email. KVM completely lacks any links related to security. Qemu has a strong link with Xen & KVM which rely on Qemu in one way or another.
While we do not offer KVM as a package, we do at present have four different versions of Xen in our tree and Qemu. This becomes a bit of a timesink when there are multiple advisories to address.

Commercial Repositories:
There are opensource projects with no publicly accessible source code repo. This makes the evaluation of the range of effected verisons difficult if the project only chooses to cover their supported versions.
ISC up until recently (past two years?) required paid membership to access BIND’s repo.
In the talk I refered to the ICU project here, this was incorrect. ICU advisories are either reserved or the bug report access blocked from public view.

OpenSSL:
Relationships with projects are important and they play a critical role in not only sharing information but code as well. Changes for 3rd party software really needs to be passed to the 3rd party to take care of. If relationships have tourned sour, it makes sharing somewhat difficult and has further implications when developing a project with the support for other projects in mind.
The LibreSSL project published a patches page which covered the changes needed to get affected software built but also co-ordinated with upstream projects to get the fixes integrated. Some projects needed more pressure^Wpersuasion than others to accept the patches.

Key components & deadware:
As mentioned previously, you want changes to go back up stream and not to carry changes in your own tree. But there are scenarios where this is possible, for example, the project is no longer developed. This is a huge problem if the project is widely used because you end up carrying local patches which hinders progress when auditing for vulnerabilites by consumers of the software downstream. It’s no longer a case of ensuring you have a specific version number but which patches are also applied to that baseline version, that then opens further questions about the patches, have you created a new issue that didn’t exist previously??
The widely used unzip utility is such an example,  are you patched for CVE-2015-7696?

An example fragmentation of fixes being carried locally is libwmf, with the announcement of some CVEs earlier this year, Jason Unovitch from the FreeBSD project discovered that there were unpatched vulnerabilities in this library going back to 2004, with patches spread across different Linux distributions, none carrying fixes for all advisories, in one case a hunk of the patch didn’t even apply. Development for libwmf stopped in the early 2000’s but it still exists as a project on sourceforge.

Jasper is another commonly used graphics library, this time for jpeg-2000, again development ceased long ago. In this case Slackware put out an advisory for their package to cover vulnerabilities from the past, going back to 2008, at which point we realised that we didn’t have the fixes either. The version in OpenBSD ports was vulnerable to the issues listed from 2014 but the vulnerabilities from prior (2008) were fixed because they’d been flagged up by the compiler in OpenBSD.

Widely Deployed:
Popular projects which have a large install base can greatly increase impact of a mistake, hence local changes should be kept to a minimum to ease maintanence and auditability.
Projects can see a very fast release cycle, especially ones which have advisories published about them regularly.
Keeping local changes to a minimum reduces the necessary effort to update. With projects which rely on downstream consumers to publish information it makes the process more difficult. Both KVM & QEMU projects do not publish any advisories themselves, at best you may have a git commit email which may be the patch you carry locally. Thankfully the Xen project publish advisories on Qemu as it can be a dependency. They are able to flesh out the details of the issue a little better than a vague commit message.
I’m unsure what happens if you’re not a Linux distro and utilise KVM.

Co-ordinating with upstream:
As I mentioned, relationships are important. An understanding and tolerrance for difference is absolutely essential in the world of software just as it is in day to day life. A common topic of disagreement is licensing, the terms expressed by said licenses and the strong opinions expressed by the participants in the disagreement. Whatever ones belief, the need to co-ordinate with people from different groups is absolutely necessary.
Of the fixes upstreamed from LibreSSL, the author of stunnel rejected a fix initially but eventually changed his mind. The change in question was a 2 line addition to add an ifdef statement so that RAND_egd function was only used if the SSL library being linked to offered such a function (detected by autoconf already). The author rejected the change based on terms of licensing of his project when the change was submitted.

Taking bigger leaps in a software project by trying to clean up a popular target can amass a large collection of local patches which need to make their way upstream. As observed by the Alpine Linux project, a Linux distribution with a new libc called musl libc. While it’s possible to build over 13000 packages with Debian 8 on pkgsrc, the package count is less than 9000 on Alpine, despite both being Linux distributions.

The submission process can be quite daunting depending on the project, to filter out submissions which may not be sound and reduce the workload of developers working on a project, some opt to requiring certain things such as results from a test suite or alike. It doesn’t help matters if project has multiple branches developed in parallel without changes being in sync.
Dealing with GNU toochain such as GCC can be very much like this. Again, local changes amass, slow transforming the local version of the toolchain to an extended version of upstream. While the toolchain may offer security features such as SSP (stack smashing protection), it’s not just the simple case of being able to switch it on, in some cases it either doesn’t work or worse, it results in broken binaries. Work to enable some of these features in pkgsrc began in the summer.

While not specifically security related, I was reminded of an incident on the OpenBSD mailing lists with the author of the once popular ion window manager. The ion developer was frustrated with operating system projects packaging older versions of his software with local changes as users where following up with him for support (in this case he was referring to RedHat). So he came on to the OpenBSD mailing list to ask about updating the software to the latest version available but that was not possible due to incompatible license changes. As usual, the combination of making demanding + licensing drama didn’t work out to well.

Conclusion

  • Organise the information on your site, for an Open Source software project your source control repo should be a single click away from your websites main page.
  • Provide a dedicated security page or section where you post advisories.
  • Write descriptive commit messages.
  • Participate in the community and help your neighbours
  • Even if you stop developing software, the code may live on longer than envisioned, think about what happens if/when you decide to stop (who become the authoritative repo).
  • Don’t make changes locally which do not go upstream by default, for it’ll surely bite you or a member of the project later down the line.
  • Publish actual advisories for your project, don’t pass the buck.
  • Technical problems are best solved with technical solutions eg a bug can still continue to exist despite adhering to a license.
  • Make the submission process to your project effortless for both parties, not just one or the other.

A week of pkgsrc #7

Time again to write up what’s been happening on the pkgsrc front since last time, this time the focus hasn’t been so much around pkgsrc on Darwin/PowerPC but more about pkgsrc in general & not necessarily code related.
At the end of the last post I mentioned a gentleman who’d been working on pkgsrc/Haiku and posting videos of his progress, I managed to make contact with him (James) & discussed his work that he’d been doing on pkgsrc. He sent me copy of the repo he’d be working off so I could assist with the aim of getting everything upstreamed as in the current state everything would need to be reintegrated per quarterly release rather than only having to pay attention if a new issue has arisen.
After getting the correct version of Haiku installed in virtualbox, I discovered a nasty bug in the Haiku network kit, it was unable to detect when the end of the file had been reached & would continue (restart?), this was revealed when I tried to download the pkgsrc tar ball from via WebPositive, ftp from the terminal was not affected however. pkgsrc bootstrapped unprivileged without issue. Hint: use the nightly snapshots until there is a newer release than Alpha 1 available.
The integration of pkgsrc into the user-land on Haiku is not currently possible due to the way the user-land is constructed, from what I understood, each Haiku package contains a piece of the filesystem, all the packages are union mounted to construct the user-land dynamically when the system comes up. That aside, with my system bootstrapped, I attempted to build Perl and ran into another bug, it seems that the library path for libperl is not populated on haiku hence perl is able to “build” but unable to run, the workaround for this in the tree I was given was to symlink libperl into ~/pkg/lib & move on. I tried various things but was unsuccessful, I believe the problem is pkgsrc specific as the version of Perl available in haikuports do not need any special treatment and the rpath is passed in correctly.
The problem was trying to isolate the required change to fix the problem, whereas in pkgsrc a policy file is passed to the build to set how Perl should be built, haikuports clobbers the source & patches in a replacement, I stopped at that point.

Haiku nightly running in virtualbox

At around about this time I received the good news about the NetBSD Foundation membership and commit bit so my focus moved to reading the various developer documentation & getting familiar with processes.

sevan.mit.edu finished a bulkbulid attempt of the entire tree which took the longest time so far to complete, through all the build attempts I uncovered a new bug, the range to use for numerical IDs of UID/GID, is not sufficient to cover all the packages in the tree that need to create an account. On further discussion with asau@, it was suggested the IDs are allocated randomly and should be fixed for consistency across builds. I started doing bulkbuilds of the entire tree on FreeBSD 10.1-RELEASE and stumbled across a very nasty bug. There is a version of tcsh package in the pkgsrc tree called shells/standalone-tcsh, this is tcsh built as a static binary & set to install to /bin (the only package in the pkgsrc tree which violates the rules and places files outside of $PREFIX by default?), this ended up overwriting the system bundled version of tcsh in FreeBSD & then deleting /bin/tcsh when the package is removed, this was fixed promptly by dholland@. It was also discovered that Python’s configure script had not been changed when FreeBSD switched to elf binaries and so still trimmed the name of libraries to account for the old linker which could not handle a minor version in a libraries filename (libpython.so.1, not libpython.so.1.0). All versions of Python had been patched in the pkgsrc tree to remove this change so that it used a consistent naming convention across all platforms. After discussing this with bapt@ (FreeBSD) at FOSDEM, it turned out to be bug and should be fixed in future Python versions once the fixes are upstreamed.

Koobs@ (FreeBSD) dug out the commit which introduced the change & the bug report.

The opportunity to join the pkg-security team came up & for the past few weeks I’ve been getting familiar with the processes of dealing with security advisories & listing them so that users who fetch the pkg-vulnerabilities database are notified if they have any vulnerable packages installed. The general advisory process is a little infuriating, based on my recent experience I’d say at the top of my list are the Oracle security advisories as they do not divulge any details other than “unknown” in version(s) X, PHP for the frequency, OpenSSL for the impact. On the one hand I was quite impressed that CVE IDs were becoming so familiar that I could spot, on the fly, an advisory that had been accounted for, but on the other hand quite upset that I was using brain capacity on this. The availability of information is quite frustrating too, issues which are assigned an ID but cannot by checked on Mitre’s site take extra effort to find the necessary information to include (Mitre are responsible for allocating the CVEs!), I should note that this is from public advisories, say from a distribution. Example, CVE-2015-0240 was announced today, the Redhat security team published a blog post covering the issue, the Mitre site at present says:
“** RESERVED ** This candidate has been reserved by an organisation or individual that will use it when announcing a new security problem. When the candidate has been publicised, the details for this candidate will be provided.”
The wording & the lack of information can also be frustrating because it’s not clear what is affected. Looking at it positively, the requirement for clarification on these discrepancies means I get lots of opportunities to approach new people in different communities to ask questions.

I created a new wiki article on the NetBSD wiki to start documenting the bootstrap process of pkgsrc on Solarish, Illumos based distributions. At present the article covers what’s required to bootstrap successfully on OmniOS, Tribblix, OpenIndiana and OpenSXCE.

One thing that’s clearly evident is my workflow needs attention, at the moment things are very clumsy, involving lots of switching around but hopefully that will be addressed in the coming month. The first thing I’ve done is setup templates for emails with the correct preferences specified so that I just need to fill the necessary information & hit send, the necessary settings are automatically applied. Still thinking about how to deal with the scenario where the system that work is being carried out on is different to the system where the patch is going to be committed from, this also happens to be a different system which a developer is using. How to deal with that in as few steps from reading, say a bug report, to generating a patch, testing it & committing a fix.

For testing patches on Mac OS X, I revisited running OS X as guest on a Mac running OS X with virtualbox. Attempts in the past had not been successful & it seemed from search results that the only approach taken was to use modified OS images for hackintosh which I did not want to take. I have a genuine machine & genuine license, I shouldn’t have to resort to 3rd party images to run this. After whinging on twitter & referencing some older links I was able to successfully virtualise Mac OS X 10.7 to 10.10 in virtualbox. Will follow up with the details on that in a separate post.

OS X Lion as a virtualbox guest

Issuing secure erase ATA command using camcontrol(8)

The ATA command set has a command to instruct a device to secure erase itself.
Depending on the application & level of sensitivity of the data on disk, it can be a convenient way to decommission a disk or reset an SSD to regain performance. On FreeBSD this can be issued using camcontrol(8).

The command below performs an enhanced erase with a timeout of 60 seconds for the command to be accepted by the disk, this is needed if you get timeout errors when you do not specify it.
camcontrol security ada0 -U user -s Erase -h Erase -T 60

Book review : Kerberos, The definitive guide

Kerberos & AFS have been two technologies I’ve wanted to deploy for a long time but based on my experience with Kerberos in Windows 2000 & and studies for MCSE I had made myself believe that it would be a painful task, I purchased this book a couple of years back but never got around to reading it properly until the start of the new year. The book is divided into 10 chapters, the first 3 explain how Kerberos works conceptually, from there on the book covers the practical aspects, how to deploy Kerberos using the MIT, Heimdal & Windows implementation, how to troubleshoot common issues, improve security, integrate applications & services, implement cross realm authentication, windows & UNIX integration, finishing off with the future of Kerberos.
The book uses FreeBSD as the OS which the UNIX examples are demonstrated on though Kerberos is built from source. I also used FreeBSD to perform my test installation but instead opted to use the Heimdal implementation which comes bundled as standard in the base OS of the BSDs. Implementation was really simple, once the KDC was up & the necessary SRV records were in place, telnet authentication worked seamlessly and after I’d set GSSAPIAuthentication yes in my ssh(1) & sshd(8) config files, SSH also worked seamlessly. Only thing that caught me out was Heimdal in FreeBSD base uses DNS where as the book assumes that this is switched off (not sure if this feature was switched off by default at the time & has now changed or it’s just the FreeBSD bundled version which has it on by default). The information for troubleshooting & some of security is still relevant but other than that it is badly outdated, discussing DES encryption & the lack of support for RC4 encryption which was the default cipher used by Windows 2000. Setting up a slave KDC has also change since this book was published in Heimdal, you now need a hprop/hostname principal for each slave server where as the book recommends host/hostname principals which doesn’t work with Heimdal anymore.

Looking around, you will still see references to Windows 2000 when doing Kerberos implementation eg in the current Heimdal documentation, I’m not sure if this is still applicable to the latest version of Windows or it’s there for historical reference.
If I were looking to learn about Kerberos, specifically Heimdal, I would use the official documentation & the Kerberos5 article on the FreeBSD handbook instead of buying this book as there is too much outdated advice in this book that no longer applies.
Ignoring the outdated best practices, the initial implementation information has remained the same over the year & it’s amazingly easy to deploy in a lab scenario for testing.

Plugins which improve the basic security of a WordPress instance

I installed a couple of plugins on my instances of wordpress to offer some basic protection which is not available in a stock wordpress install.
First plugin is Simple Security, this plugin protects against brute force login attempts. (There seems to be an issue with the current 1.0.3 release of this plugin where browsing multipage section of the admin section eg posts, followed up with the developer for feedback).

Second plugin is Ban Hammer which allows you to block accounts from signing up using a listed domain or specific email address.