December 08, 2016

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppAPT 0.0.3

A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt, apt-get, apt-cache, ... commands and their cache powering Debian, Ubuntu and the like -- is now on CRAN.

We changed the package to require C++11 compilation as newer Debian systems with g++-6 and the current libapt-pkg-dev library cannot build under the C++98 standard which CRAN imposes (and let's not get into why ...). Once set to C++11 we have no issues. We also added more examples to the manual pages, and turned on code coverage.

A bit more information about the package is available here as well as as the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

08 December, 2016 01:19AM

December 07, 2016

hackergotchi for Shirish Agarwal

Shirish Agarwal

Day trip in Cape Town, part 2

Debconf16 logo

The post continues from the last post shared.

Let me get some interesting tit-bits not related to the day-trip out-of-the-way first –

I don’t know whether we had full access to see all parts of fuller hall or not. Couple of days I was wondering around Fuller Hall, specifically next to where clothes were pressed. Came to know of the laundry service pretty late but still was useful. Umm… next to where the ladies/gentleman pressed our clothes, there is a stairway which goes down. In fact even on the opposite side there is a stairway which goes down. I dunno if other people explored them or not.

The jail inside and under UCT

I was surprised and shocked to see bars in each room as well as connecting walkways etc. I felt a bit sad, confused and curious and went on to find more places like that. After a while I came up to the ground-level and enquired with some of the ladies therein. I was shocked to know that UCT some years ago (they were not specific) was a jail for people. I couldn’t imagine that a place which has so much warmth (in people, not climate) could be ‘evil’ in a sense. I was not able to get much information out of them about the nature of jail it was, maybe it is a dark past that nobody wants to open up, dunno. There were also two *important* aspects of UCT which Bernelle either forgot, didn’t share or I just came to know via the Wikipedia page then but nothing else.

1. MeerKAT – Apparently quite a bit of the technology was built-in UCT itself. This would have been interesting for geeks and wanna-be geeks like me🙂

2. The OpenContent Initiative by UCT – This would have been also something worth exploring.

One more interesting thing which I saw was the French council in Cape Town from outside

The French Council in cape town from outside

I would urge to look at the picture in the gallery as the picture I shared doesn’t really show all the details. For e.g. the typical large french windows which are the hall-mark of French architecture doesn’t show its glory but if you look at 1306×2322 original picture instead of the 202×360 reproduction you will see that.

You will also the insignia of the French Imperial Eagle whose history I came to know only after I looked it up on the Wikipedia page on that day.

It seemed fascinating and probably would have the same pride as the State Emblem of India has for Indians with the four Asiatic Lions standing in a circle protecting each other.

I also like the palm tree and the way the French Council seemed little and yet had character around all the big buildings.

What also was interesting that there wasn’t any scare/fear-build and we could take photos from outside unlike what I had seen and experienced in Doha, Qatar as far as photography near Western Embassies/Councils were concerned.

One of the very eye-opening moments for me was also while I was researching flights from India to South Africa. While perhaps unconsciously I might have known that Middle East is close to India, in reality, it was only during the search I became aware that most places in Middle East by flight are only an hour or two away.

This was shocking as there is virtually no mention of one of our neighbours when they are source of large-scale remittances every year. I mean this should have been in our history and geography books but most do not dwell on the subject. It was only during and after that I could understand Mr. Modi’s interactions and trade policies with the Middle East.

Another interesting bit was seeing a bar in a Sprinbok bus –

spingbok atlas bar in bus

While admittedly it is not the best picture of the bar, I was surprised to find a bar at the back of a bus. By bar I mean a machine which can serve anything from juices to alcoholic drinks depending upon what is stocked. What was also interesting in the same bus is that the bus also had a middle entrance-and-exit.

The middle door in springbok atlas

This is something I hadn’t seen in most Indian buses. Some of the Volvo buses have but it is rarely used (only except emergencies) . An exhaustive showcase of local buses can be seen here . I find the hand-drawn/cad depictions of all the buses by Amit Pense near to the T.

Axe which can be used to break windows

Emergency exit window

This is also something which I have not observed in Indian inter-city buses (axe to break the window in case of accident and breakable glass which doesn’t hurt anyone I presume), whether they are State-Transport or the high-end Volvo’s . Either it’s part of South African Roads Regulations or something that Springbok buses do for their customers. All of these queries about the different facets I wanted to ask the bus-driver and the attendant/controller but in the excitement of seeing, recording new things couldn’t ask😦

In fact one of the more interesting things I looked at and could look day and night is the variety of vehicles on display in Cape Town. In hindsight, I should have bought a couple of 128 GB MMC cards for my mobile rather than the 64 GB one. It was just plain inadequate to capture all that was new and interesting.

Auditorum chair truck seen near Auditorium

This truck I had seen about some 100 metres near the Auditorium on Upper Campus. The truck’s design, paint was something I had never seen before. It is/was similar to casket trucks seen in movies but the way it was painted and everything made it special.

What was interesting is to see the gamut of different vehicles. For instance, there were no bicycles that I saw in most places. There were mostly Japanese/Italian bikes and all sorts of trucks. If I had known before, I would definitely have bought an SD specifically to take snaps of all the different types of trucks, cars etc. that I saw therein.

The adage/phrase ” I should stop in any one place and the whole world will pass me by ” seemed true on quite a few South African Roads. While the roads were on par or a shade better than India, many of those were wide roads. Seeing those, I was left imagining how the Autobahn in Germany and other high-speed Expressways would look in feel.

India has also been doing that with the Pune-Mumbai Expressway and projects like Yamuna Expressway and now the extension Agra Lucknow Expressway but doing this all over India would take probably a decade or more. We have been doing it since a decade and a half. NHDP and PMGSY are two projects which are still ongoing to better the roads. We have been having issues as to should we have toll or no toll issues but that is a discussion for some other time.

One of the more interesting sights I saw was the high-arched gothic-styled church from outside. This is near Longstreet as well.

high arch gothic-styled church

I have seen something similar in Goa, Pondicherry but not such high-arches. I did try couple of times to gain entry but one time it was closed, the other time some repairing/construction work was going on or something. I would loved to see it from inside and hopefully they would have had an organ (music) as well. I could imagine to some extent the sort of music that would have come out.

Now that Goa has come in the conversation I can’t help but state that Seafood enthusiasts/lover/aficionado, or/and Pescatarianism would have a ball of a time in Goa. Goa is on the Konkan coast and while I’m veggie, ones who enjoy seafood really have a ball of a time in Goa. Fouthama’s Festival which happens in February is particularly attractive as Goan homes are thrown open for people to come and sample their food, exchange recipes and alike. This happens around 2 weeks before the Goan Carnival and is very much a part of the mish-mashed Konkani-Bengali-Parsi-Portugese culture.

I better stop here about the Goa otherwise I’ll get into reminiscing mode.

To put the story and event back on track from where we left of (no fiction hereon), Nicholas was in constant communication with base, i.e. UCT as well as another group who was hiking from UCT to Table Mountain. We waited for the other group to join us then at 13:00 hrs. it was decided to move along as they had lost their way somewhere in-between .

We came down the same cable-car and then ventured on towards Houtbay. Houtbay has it all, a fisherman’s wharf, actual boats with tough-mean looking men with tattoos working on boats puffing cigars/pipes, gaggle of sea-gulls, the whole scene. Sharing a few pictures of the way in-between.

the view en-route to Houtbay

western style car paint and repair shop

Tajmahal Indian Restaurant, Houtbay

I just now had a quick look at the restaurant and it seems they had options for veggies too. Unfortunately, the rating leaves a bit to be desired but then dunno as Indian flavoring is something that takes time to get used too. Zomato doesn’t give any idea of from when a restaurant is in business and has too few reviews so not easy to know.

Chinese noodles and small houses

Notice the pattern, the pattern of small houses I saw all the way till Houtbay and back. I do vaguely remember starting a discussion about it but don’t really remember. I have seen (on TV) cities like Miami, Dubai or/and Hong Kong who have big buildings but both in Konkan as well as Houtbay there were small buildings. I guess a combination of zoning regulations, feel of community, fear of being flooded all play into beaches being the way they are.

Also, this probably is good as less stress on the environment.

Miamiboyz from Wikimedia Commons

Audi rare car to be seen in India

The Audi – rare car to be seen in India. This car has been associated with Ravi Shastri when he won it in 1985. I was young but still get goosebumps remembering those days.

first-glance-Houtbay-and-pier

First glance of Houtbay beach and pier. Notice how clean and white the beach is.

Wharf-Grill-Restaurant-from-side-and-Hop-on-Hop-off-bus

You can see the wharf grill restaurant in the distance (side-view), see the back of the hop on and hop off bus (a concept which was unknown to me till then). Once I came back and explored came to know this concept is prevalent in many a touristy places around the world. Umm… also By sheer happenchance also captured a beautiful looking Indian female😉 .

So many things happening all at once

In Hindi, we would call this picture ‘virodabhas’ or ‘contradiction’. this is in afternoon, around 1430 hrs. You have the sun, the clouds, the Mountains, the x number of boats, the pier, the houses, the cars, the shops. It was all crazy and beautiful at the same time. The Biggest Contradiction is seeing the Mountain, the beach and the Sea in the same Picture. Baffles the mind.

We were supposed to go on a short cruise to seal/dolphin island but as we were late (as had been waiting for the other group) didn’t go and instead just loitered there.

Fake-real lookout bar-restaurant

IIRC the lookout bar is situated just next to Houtbay Search and Rescue. Although was curious if the Lookout tower was used in case of disappearances.

Seal in action

Seal jumping over water, what a miracle !

One of the boats on which we possibly could have been on.

It looked like the boat we could have been on. I clicked as I especially liked the name Calypso and Calypso . I shared the two links as the mythologies, interpretation differ a bit between Greek and Hollywood culture🙂

Debian folks and the area around

Can see few Debian folks in the foreground, next to the Pole and the area around. Also can see a bit of the area around.

Alone boy trying to surf

I don’t know anything about water sports and after sometime he came out. I was left wondering though, how safe he was in that water. While he was close to the pier and he was just paddling, there weren’t big waves still felt a bit of concern.

Mr. Seal - the actor and his handler

While the act was not to the level we see in the movies, still for the time I hung around, I saw him showing attitude for his younger audiences, eating out of their hands, making funny sounds. Btw he farted a few times, whether that was a put-on or not can’t really say but produced a few guffaws from his audience.

A family feeding Mr. Seal

I dunno why the birds came down for. Mr. Seal was being fed oily small fish parts, dunno if the oil was secreted by the fish themselves or whatever, it just looked oily from distance.

Bird-Man-Bird

Bird taking necessary sun bath

typical equipment on a boat to catch fish-lot of nets

boats-nets-and-ropes

People working on disentangling a net

There wasn’t much activity on the time we went. It probably would have been different on sunrise and would be on sunset. The only activity I saw was on this boat where they were busy fixing and disentangling the lines. I came up with 5-15 different ideas for a story but rejected them as –

a. Probably all of them have been tried. People have been fishing since the beginning of time and modern fishing probably 200 odd years or so. I have read accounts of fishing companies in early 1800s onwards, so probably all must have been tried.

b. More dangerous one, if there is a unique idea, then it becomes more dangerous as writing is an all-consuming process. Writing a blog post (bad or good) takes lots of time. I constantly read, re-read, try and improvise till I can or my patience loses out. In book you simply can’t have such luxuries.

hout-bay-search-and-rescue-no-parking-zone

No parking/tow zone in/near the Houtbay search and rescue. Probably to take out emergency vehicles once something untoward happens.

hout-bay-sea-rescue-with-stats

Saved 54 lives, boats towed 154 – Salut! Houtbay sea rescue.

The different springbok atlas bus that we were on

kraal-kraft

The only small criticism is for Houtbay – there wasn’t a single public toilet. We had to ask favor at kraal kraft to use their toilets and there could have been accidents, it wasn’t lighted well and water was spilled around.

Road sign telling that we are near to UCT

For us, because we were late we missed both the boat-cruise as well as some street shops selling trinkets. Other than that it was all well. We should have stayed till sunset, I am sure the view would have been breath-taking but we hadn’t booked till evening.

Back at UCT

Overall it was an interesting day as we had explored part of Table Mountain, seen the somewhat outrageously priced trinkets there as well as explored Houtbay sea-side as well.


Filed under: Miscellenous Tagged: #Audi, #Cape Town, #Cruises, #Debconf16, #French Council, #Geography, #Houtbay Sea Rescue, #Jail, #Middle East, #Springbok Atlas, #Vehicles

07 December, 2016 09:10PM by shirishag75

Tianon Gravi

My Docker Install Process

I’ve had several requests recently for information about how I personally set up a new machine for running Docker (especially since I don’t use the infamous curl get.docker.com | sh), so I figured I’d outline the steps I usually take.

For the purposes of simplicity, I’m going to assume Debian (specifically stretch, the upcoming Debian stable release), but these should generally be easily adjustable to jessie or Ubuntu.

These steps should be fairly similar to what’s found in upstream’s “Install Docker on Debian” document, but do differ slightly in a few minor ways.

grab Docker’s APT repo GPG key

The way I do this is probably a bit unconventional, but the basic gist is something like this:

export GNUPGHOME="$(mktemp -d)"
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
gpg --export --armor 58118E89F3A912897C070ADBF76221572C52609D | sudo tee /etc/apt/trusted.gpg.d/docker.gpg.asc
rm -rf "$GNUPGHOME"

(On jessie or another release whose APT doesn’t support .asc files in /etc/apt/trusted.gpg.d, I’d drop --armor and the .asc and go with simply /.../docker.gpg.)

This creates me a new GnuPG directory to work with (so my personal ~/.gnupg doesn’t get cluttered with this new key), downloads Docker’s signing key from the keyserver gossip network (verifying the fetched key via the full fingerprint I’ve provided), exports the key into APT’s keystore, then cleans up the leftovers.

For completeness, other popular ways to fetch this include:

sudo apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

(worth noting that man apt-key discourages the use of apt-key adv)

wget -qO- 'https://apt.dockerproject.org/gpg' | sudo apt-key add -

(no verification of the downloaded key)

Here’s the relevant output of apt-key list on a machine where I’ve got this key added in the way I outlined above:

$ apt-key list
...

/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub   rsa4096 2015-07-14 [SCEA]
      5811 8E89 F3A9 1289 7C07  0ADB F762 2157 2C52 609D
uid           [ unknown] Docker Release Tool (releasedocker) <docker@docker.com>

...

add Docker’s APT source

If you prefer to fetch sources via HTTPS, install apt-transport-https, but I’m personally fine with simply doing GPG verification of fetched packages, so I forgo that in favor of less packages installed. YMMV.

echo 'deb http://apt.dockerproject.org/repo debian-stretch main' | sudo tee /etc/apt/sources.list.d/docker.list

Hopefully it’s obvious, but debian-stretch in that line should be replaced by debian-jessie, ubuntu-xenial, etc. as desired. It’s also worth pointing out that this will not include Docker’s release candidates. If you want those as well, add testing after main, ie ... debian-stretch main testing' | ....

At this point, you should be safe to run apt-get update to verify the changes:

$ sudo apt-get update
...
Hit:1 http://apt.dockerproject.org/repo debian-stretch InRelease
...
Reading package lists... Done

(There shouldn’t be any warnings or errors about missing keys, etc.)

configure Docker

This step could be done after Docker’s installed (and indeed, that’s usually when I do it because I forget that I should until I’ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn’t have to be restarted later.

sudo mkdir -p /etc/docker
sudo sensible-editor /etc/docker/daemon.json

(sensible-editor can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default)

I then fill daemon.json with at least a default storage-driver. Whether I use aufs or overlay2 depends on my kernel version and available modules – if I’m on Ubuntu, AUFS is still a no-brainer (since it’s included in the default kernel if the linux-image-extra-XXX/linux-image-extra-virtual package is installed), but on Debian AUFS is only available in either 3.x kernels (jessie’s default non-backports kernel) or recently in the aufs-dkms package (as of this writing, still only available on stretch and sid – no jessie-backports option).

If my kernel is 4.x+, I’m likely going to choose overlay2 (or if that errors out, the older overlay driver).

Choosing an appropriate storage driver is a fairly complex topic, and I’d recommend that for serious production deployments, more research on pros and cons is performed than I’m including here (especially since AUFS and OverlayFS are not the only options – they’re just the two I personally use most often).

{
	"storage-driver": "overlay2"
}

configure boot parameters

I usually set a few boot parameters as well (in /etc/default/grub’s GRUB_CMDLINE_LINUX_DEFAULT option – run sudo update-grub after adding these, space-separated).

  • cgroup_enable=memory – enable “memory accounting” for containers (allows docker run --memory for setting hard memory limits on containers)
  • swapaccount=1 – enable “swap accounting” for containers (allows docker run --memory-swap for setting hard swap memory limits on containers)
  • systemd.legacy_systemd_cgroup_controller=yes – newer versions of systemd may disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see systemd/systemd#4628, opencontainers/runc#1175, docker/docker#28109)
  • vsyscall=emulate – allow older binaries to run (debian:wheezy, etc.; see docker/docker#28705)

All together:

...
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"
...

install Docker!

Finally, the time has come.

$ sudo apt-get install -V docker-engine
...

$ sudo docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:45:16 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:45:16 2016
 OS/Arch:      linux/amd64

$ sudo usermod -aG docker "$(id -un)"

(Reboot or logout/login to update your session to include docker group membership and thus no longer require sudo for using docker commands.)

Hope this is useful to someone! If nothing else, it’ll serve as a concise single-page reference for future-tianon. 😇

07 December, 2016 07:00AM by Tianon Gravi (admwiggin@gmail.com)

Jonas Meurer

On CVE-2016-4484, a (securiy)? bug in the cryptsetup initramfs integration

On CVE-2016-4484, a (security)? bug in the cryptsetup initramfs integration

On November 4, I was made aware of a security vulnerability in the integration of cryptsetup into initramfs. The vulnerability was discovered by security researchers Hector Marco and Ismael Ripoll of CyberSecurity UPV Research Group and got CVE-2016-4484 assigned.

In this post I'll try to reflect a bit on

What CVE-2016-4484 is all about

Basically, the vulnerability is about two separate but related issues:

1. Initramfs rescue shell considered harmful

The main topic that Hector Marco and Ismael Ripoll address in their publication is that Debian exits into a rescue shell in case of failure during initramfs, and that this can be triggered by entering a wrong password ~93 times in a row.

Indeed the Debian initramfs implementation as provided by initramfs-tools exits into a rescue shell (usually a busybox shell) after a defined amount of failed attempts to make the root filesystem available. The loop in question is in local_device_setup() at the local initramfs script

In general, this behaviour is considered as a feature: if the root device hasn't shown up after 30 rounds, the rescue shell is spawned to provide the local user/admin a way to debug and fix things herself.

Hector Marco and Ismael Ripoll argue that in special environments, e.g. on public computers with password protected BIOS/UEFI and bootloader, this opens an attack vector and needs to be regarded as a security vulnerability:

It is common to assume that once the attacker has physical access to the computer, the game is over. The attackers can do whatever they want. And although this was true 30 years ago, today it is not.

There are many "levels" of physical access. [...]

In order to protect the computer in these scenarios: the BIOS/UEFI has one or two passwords to protect the booting or the configuration menu; the GRUB also has the possibility to use multiple passwords to protect unauthorized operations.

And in the case of an encrypted system, the initrd shall block the maximum number of password trials and prevent the access to the computer in that case.

While Hector and Ismael have a valid point in that the rescue shell might open an additional attack vector in special setups, this is not true for the vast majority of Debian systems out there: in most cases a local attacker can alter the boot order, replace or add boot devices, modify boot options in the (GNU GRUB) bootloader menu or modify/replace arbitrary hardware parts.

The required scenario to make the initramfs rescue shell an additional attack vector is indeed very special: locked down hardware, password protected BIOS and bootloader but still local keyboard (or serial console) access are required at least.

Hector and Ismael argue that the default should be changed for enhanced security:

[...] But then Linux is used in more hostile environments, this helpful (but naive) recovery services shall not be the default option.

For the reasons explained about, I tend to disagree to Hectors and Ismaels opinion here. And after discussing this topic with several people I find my opinion reconfirmed: the Debian Security Team disputes the security impact of the issue and others agree.

But leaving the disputable opinion on a sane default aside, I don't think that the cryptsetup package is the right place to change the default, if at all. If you want added security by a locked down initramfs (i.e. no rescue shell spawned), then at least the bootloader (GNU GRUB) needs to be locked down by default as well.

To make it clear: if one wants to lock down the boot process, bootloader and initramfs should be locked down together. And the right place to do this would be the configurable behaviour of grub-mkconfig. Here, one can set a password for GRUB and the boot parameter 'panic=1' which disables the spawning of a rescue shell in initramfs.

But as mentioned, I don't agree that this would be sane defaults. The vast majority of Debian systems out there don't have any security added by locked down bootloader and initramfs and the benefit of a rescue shell for debugging purposes clearly outrivals the minor security impact in my opinion.

For the few setups which require the added security of a locked down bootloader and initramfs, we already have the relevant options documented in the Securing Debian Manual:

After discussing the topic with initramfs-tools maintainers today, Guilhem and me (the cryptsetup maintainers) finally decided to not change any defaults and just add a 'sleep 60' after the maximum allowed attempts were reached.

2. tries=n option ignored, local brute-force slightly cheaper

Apart from the issue of a rescue shell being spawned, Hector and Ismael also discovered a programming bug in the cryptsetup initramfs integration. This bug in the cryptroot initramfs local-top script allowed endless retries of passphrase input, ignoring the tries=n option of crypttab (and the default of 3). As a result, theoretically unlimited attempts to unlock encrypted disks were possible when processed during initramfs stage. The attack vector here was that local brute-force attacks are a bit cheaper. Instead of having to reboot after max tries were reached, one could go on trying passwords.

Even though efficient brute-force attacks are mitigated by the PBKDF2 implementation in cryptsetup, this clearly is a real bug.

The reason for the bug was twofold:

  • First, the condition in setup_mapping() responsible for making the function fail when the maximum amount of allowed attempts is reached, was never met:

    setup_mapping()
    {
      [...]
      # Try to get a satisfactory password $crypttries times
      count=0                              
    while [ $crypttries -le 0 ] || [ $count -lt $crypttries ]; do export CRYPTTAB_TRIED="$count" count=$(( $count + 1 )) [...] done if [ $crypttries -gt 0 ] && [ $count -gt $crypttries ]; then message "cryptsetup: maximum number of tries exceeded for $crypttarget" return 1 fi [...] }

    As one can see, the while loop stops when $count -lt $crypttries. Thus the second condition $count -gt $crypttries is never met. This can easily be fixed by decreasing $count by one in case of a successful unlock attempt along with changing the second condition to $count -ge $crypttries:

    setup_mapping()
    {
      [...]
      while [ $crypttries -le 0 ] || [ $count -lt $crypttries ]; do
          [...]
          # decrease $count by 1, apparently last try was successful.
          count=$(( $count - 1 ))
          [...]
      done
      if [ $crypttries -gt 0 ] && [ $count -ge $crypttries ]; then
          [...]
      fi
      [...]
    }
    

    Christian Lamparter already spotted this bug back in October 2011 and provided a (incomplete) patch, but back then I even managed to merge the patch in an improper way, making it even more useless: The patch by Christian forgot to decrease $count by one in case of a successful unlock attempt, resulting in warnings about maximum tries exceeded even for successful attemps in some circumstances. But instead of adding the decrease myself and keeping the (almost correct) condition $count -eq $crypttries for detection of exceeded maximum tries, I changed back the condition to the wrong original $count -gt $crypttries that again was never met. Apparently I didn't test the fix properly back then. I definitely should do better in future!

  • Second, back in December 2013, I added a cryptroot initramfs local-block script as suggested by Goswin von Brederlow in order to fix bug #678692. The purpose of the cryptroot initramfs local-block script is to invoke the cryptroot initramfs local-top script again and again in a loop. This is required to support complex block device stacks.

    In fact, the numberless options of stacked block devices are one of the biggest and most inglorious reasons that the cryptsetup initramfs integration scripts became so complex over the years. After all we need to support setups like rootfs on top of LVM with two separate encrypted PVs or rootfs on top of LVM on top of dm-crypt on top of MD raid.

    The problem with the local-block script is that exiting the setup_mapping() function merely triggers a new invocation of the very same function.

    The guys who discovered the bug suggested a simple and good solution to this bug: When maximum attempts are detected (by second condition from above), the script sleeps for 60 seconds. This mitigates the brute-force attack options for local attackers - even rebooting after max attempts should be faster.

About disclosure, wording and clickbaiting

I'm happy that Hector and Ismael brought up the topic and made their argument about the security impacts of an initramfs rescue shell, even though I have to admit that I was rather astonished about the fact that they got a CVE assigned.

Nevertheless I'm very happy that they informed the Security Teams of Debian and Ubuntu prior to publishing their findings, which put me in the loop in turn. Also Hector and Ismael were open and responsive when it came to discussing their proposed fixes.

But unfortunately the way they advertised their finding was not very helpful. They announced a speech about this topic at the DeepSec 2016 in Vienna with the headline Abusing LUKS to Hack the System.

Honestly, this headline is missleading - if not wrong - in several ways:

  • First, the whole issue is not about LUKS, neither is it about cryptsetup itself. It's about Debians integration of cryptsetup into the initramfs, which is a compeletely different story.
  • Second, the term hack the system suggests that an exploit to break into the system is revealed. This is not true. The device encryption is not endangered at all.
  • Third - as shown above - very special prerequisites need to be met in order to make the mere existance of a LUKS encrypted device the relevant fact to be able to spawn a rescue shell during initramfs.

Unfortunately, the way this issue was published lead to even worse articles in the tech news press. Topics like Major security hole found in Cryptsetup script for LUKS disk encryption or Linux Flaw allows Root Shell During Boot-Up for LUKS Disk-Encrypted Systems suggest that a major security vulnerabilty was revealed and that it compromised the protection that cryptsetup respective LUKS offer.

If these articles/news did anything at all, then it was causing damage to the cryptsetup project, which is not affected by the whole issue at all.

After the cat was out of the bag, Marco and Ismael aggreed that the way the news picked up the issue was suboptimal, but I cannot fight the feeling that the over-exaggeration was partly intended and that clickbaiting is taking place here. That's a bit sad.

07 December, 2016 01:53AM

December 06, 2016

hackergotchi for Sylvain Le Gall

Sylvain Le Gall

Release of OASIS 0.4.8

I am happy to announce the release of OASIS v0.4.8.

Logo OASIS small

OASIS is a tool to help OCaml developers to integrate configure, build and install systems in their projects. It should help to create standard entry points in the source code build system, allowing external tools to analyse projects easily.

This tool is freely inspired by Cabal which is the same kind of tool for Haskell.

You can find the new release here and the changelog here. More information about OASIS in general on the OASIS website.

Pull request for inclusion in OPAM is pending.

Here is a quick summary of the important changes:

  • Fix various problems of parsing present in OASIS 0.4.7 (extraneous whitespaces, handling of ocamlbuild argument...)
  • Enable creation of OASIS plugin and OASIS command line plugin.
  • Various fixes for the plugin "omake".
  • Create 2 branches to pin OASIS with OPAM, making easier for contributor to test dev. version.

Thanks to Edwin Török, Yuri D. Lensky and Gerd Stolpmann for their contributions.

06 December, 2016 11:17PM by gildor

hackergotchi for Mirco Bauer

Mirco Bauer

Secure USB boot with Debian

Foreword

The moment you leave your laptop, say in a hotel room, you can no longer trust your system as it could have been modified while you were away. Think you are safe because you have a crypted disk? Well, if the boot partition is on the laptop itself, it can be manipulated and you will not notice because the boot partition can't be encrypted. The BIOS needs to access the MBR and boot loader and that loads the Linux kernel, all unencrypted. There has been some reports lately that the Linux cryptsetup is insecure because you can spawn a root shell by hitting the enter key for 70 seconds. This is not the real threat to your system, really. If someone has physical access to your hardware, he can get a root shell in less than a second by passing init=/bin/bash as parameter to the Linux kernel in the boot loader regardless if cryptsetup is used or not! The attacker can also use other ways like booting a live system from CD/USB etc. The real insecurity here is the unencrypted boot partition and not some script that gets executed from it. So how to prevent this physical access attack vector? Just keep reading this guide.

This guide explains how to install Debian securely on your laptop with using an external USB boot disk. The disk inside the laptop should not contain your /boot partition since that is an easy target for manipulation. An attacker could for example change the boot scripts inside the initrd image to capture your passphrase of your crypted volume. With an USB boot partition, you can unplug the USB stick after the operating system has booted. Best practice here is to have the USB stick together with your bunch of keys. That way you will disconnect your USB stick early after the boot as finished so you can put it back into your pocket.

Secure Hardware Assumptions

We have to assume here that the hardware you are using to download and verify the install media is safe to use. Same applies with the hardware where you are doing the fresh Debian install. Say the hardware does not contain any malware in the form of code in EFI or other manipulation attempts that influence the behavior of the operating system we are going to install.

Download Debian Install ISO

Feel free to use any Debian mirror and install flavor. For this guide I am using the download mirror in Germany and the DVD install flavor.

wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/debian-8.6.0-amd64-DVD-1.iso

Verify hashsum of ISO file

To know if the ISO file was downloaded without modification we have to check the hashsum of the file. The hashsum file can be found in the same directory as the ISO file on the download mirror. With hashsums if a single bit differs in the file, the resulting SHA512 sum will be completely different.

Obtain the hashsum file using:

wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/SHA512SUMS

Calculate a local hashsum from the downloaded ISO file:

sha512sum debian-8.6.0-amd64-DVD-1.iso

Now you need to compare the hashsum with that is in the SHA512SUMS file. Since the SHA512SUMS file contains the hashsums of all files that are in the same directory you need to find the right one first. grep can do this for you:

grep debian-8.6.0-amd64-DVD-1.iso SHA512SUMS

Both commands executed after each other should show following output:

$ sha512sum debian-8.6.0-amd64-DVD-1.iso
c3883edfc95e3b09152d46ce29a032eed1de71531549aee86bb98dab1528088a16f0b4d628aee8ac6cc420364e208d3d5e19d0dea3576f53b904c18e8f604d8c  debian-8.6.0-amd64-DVD-1.iso
$ grep debian-8.6.0-amd64-DVD-1.iso SHA512SUMS
c3883edfc95e3b09152d46ce29a032eed1de71531549aee86bb98dab1528088a16f0b4d628aee8ac6cc420364e208d3d5e19d0dea3576f53b904c18e8f604d8c  debian-8.6.0-amd64-DVD-1.iso

As you can see the hashsum found in the SHA512SUMS file matches with the locally generated hashsum using the sha512sum command.

At this point we are not finished yet. These 2 matching hashsums just means whatever was on the download server matches what we have received and stored locally on your disk. The ISO file and SHA512SUM file could still be a modified version!

And this is where GPG signatures chime in, covered in the next section.

Download GPG Signature File

GPG signature files usually have the .sign file name extension but could also be named .asc. Download the signature file using wget:

wget http://ftp.de.debian.org/debian-cd/current/amd64/iso-dvd/SHA512SUMS.sign

Obtain GPG Key of Signer

Letting gpg verify the signature will fail at this point as we don't have the public key of the signer:

$ gpg --verify SHA512SUMS.sign
gpg: assuming signed data in 'SHA512SUMS'
gpg: Signature made Mon 19 Sep 2016 12:23:47 AM HKT
gpg:                using RSA key DA87E80D6294BE9B
gpg: Can't check signature: No public key

Downloading a key is trivial with gpg, but more importantly we need to verify that this key (DA87E80D6294BE9B) is trustworthy, as it could also be a key of the infamous man-in-the-middle.

Here you can find the GPG fingerprints of the official signing keys used by Debian. The ending of the "Key fingerprint" line should match the key id we found in the signature file from above.

gpg:                using RSA key DA87E80D6294BE9B

Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

DA87E80D6294BE9B matches Key fingerprint = DF9B 9C49 EAA9 2984 3258 9D76 DA87 E80D 6294 BE9B

To download and import this key run:

$ gpg --keyserver keyring.debian.org --recv-keys DA87E80D6294BE9B

Verify GPG Signature of Hashsum File

Ok, we are almost there. Now we can run the command which checks if the signature of the hashsum file we have, was not modified by anyone and matches what Debian has generated and signed.

gpg: assuming signed data in 'SHA512SUMS'
gpg: Signature made Mon 19 Sep 2016 12:23:47 AM HKT
gpg:                using RSA key DA87E80D6294BE9B
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "Debian CD signing key <debian-cd@lists.debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

The important line in this output is the "Good signature from ..." one. It still shows a warning since we never certified (signed) that Debian key. This can be ignored at this point though.

Write ISO Image to Install Media

With a verified pristine ISO file we can finally start the install by writing it to an USB stick or blank DVD. So use your favorite tool to write the ISO to your install media and boot from it. I have used dd and a USB stick attached as /dev/sdb.

dd if=debian-8.6.0-amd64-DVD-1.iso of=/dev/sdb bs=1M oflag=sync

Install Debian on Crypted Volume with USB boot partition

I am not explaining each step of the Debian install here. The Debian handbook is a good resource for covering each install step.

Follow the steps until the installers wants to partition your disk.

There you need to select the "Guided, use entire disk and set up encrypted LVM" option. After that select the built-in disk of your laptop, which usually is sda but double check this before you go ahead, as it will overwrite the data! The 137 GB disk in this case is the built-in disk and the 8 GB is the USB stick.

It makes no difference at this point if you select "All files in one partition" or "Separate /home partition". The USB boot partition can be selected a later step.

Confirm that you want to overwrite your built-in disk shown as sda. It will take a while as it will write random data to the disk to ensure there is no unencrypted data left on the disk from previous installations for example.

Now you need to enter your passphrase that will be used to protect the private key of the crypt volume. Choose something long enough like a sentence and don't forget the passphrase else you can no longer access your data! Don't save the passphrase on any computer, smartphone or password manager. If you want to make a backup of your passphrase then use a ball pen and paper and store the paper backup in a secure location.

The installer will show you a summary of the partitioning as shown above but we need to make the change for the USB boot disk. At the moment it wants to put /boot on sda which is the built-in disk, while our USB stick is sdb. Select /boot and hit enter, after that select "Delete this partition".

After /boot was deleted we can create /boot on the USB stick shown as sdb. Select sdb and hit enter. It will ask if you want to create an empty partition table. Confirm that question with yes.

The partition summary shows sdb with no partitions on it. Select FREE SPACE and select "Create a new partition". Confirm the suggested partition size. Confirm the partition type to be "Primary".

It is time to tell the installer to use this new partition on the USB stick (sdb1) as /boot partition. Select "Mount point: /home" and in the next dialog select "/boot - static files of the boot loader" as shown below:

Confirm the made changes by selecting "Done setting up the partition".

The final partitioning should look now like the following screenshot:

If the partition summary looks good, go ahead with the installation by selecting "Finish partitioning and write changes to disk".

When the installer asks if it should force EFI, then select no, as EFI is not going to protect you.

Finish the installation as usual, select your preferred desktop environment etc.

GRUB Boot Loader

Confirm the dialog that wants to install GRUB to the master boot record. Here it is important to install it to the USB stick and not your built-in SATA/SSD disk! So select sdb (the USB stick) in the next dialog.

First Boot from USB

Once everything is installed, you can boot from your USB stick. As simple test you can unplug your USB stick and the boot should fail with "no operating system found" or similar error message from the BIOS. If it doesn't boot even though the USB stick is connected, then most likely your BIOS is not configured to boot from USB media. Also a blank screen and nothing happening is usually meaning the BIOS can't find a boot device. You need to change the boot setting in your BIOS. As the steps are very different for each BIOS, I can't provide a detailed step-by-step list here.

Usually you can enter the BIOS using F1, F2 or F12 after powering on your computer. In the BIOS there is a menu to configure the boot order. In that list it should show USB disk/storage as the first position. After you have made the changes save and exit the BIOS. Now it will boot from your USB stick first and GRUB will show up and proceeds with the boot process till it will ask for your passphrase to unlock the crypt volume.

Unmount /boot partition after Boot

If you boot your laptop from the USB stick, we want to remove the stick after it has finished booting. This will prevent an attacker to make modifications to your USB stick. To avoid data loss, we should not simply unplug the USB stick but unmount /boot first and then unplug the stick. Good news is that we can automate this unmounting and you just need to unplug the stick after the laptop has finished booting to your login screen.

Just add this line to your /etc/rc.local file:

umount /boot

After boot you can once verify that it automatically unmounts /boot for you by running:

mount | grep /boot

If that command produces no output, then /boot is not mounted and you can safely unplug the USB stick.

Final Words

From time to time you need to upgrade your Linux kernel of course which is on the /boot partition. This can still be done the regular way using apt-get upgrade, except that you need to mount /boot before that and unmount it again after the kernel upgrade.

Enjoy your secured laptop. Now you can leave it in a hotel room without the possibility of someone trying you obtain your passphrase by putting a key logger in your boot partition. All the attacker will see is a fully encrypted harddisk. If he tries to mess with your crypted disk, you will notice as the decryption will fail.

Disclaimer: there are still other attack vectors possible, but they are much harder to do. Your hardware or BIOS can still be modified. But not by holding down the enter key for 70 seconds or by booting a live system.

06 December, 2016 01:28PM

December 05, 2016

hackergotchi for Shirish Agarwal

Shirish Agarwal

The Anti-Pollito squad – arrest and confession

Disclaimer – This is an attempt at humor and hence entirely fictional in nature. While some incidents depicted are true, the context and the story woven around them are by yours truly. None of the Mascots of Debian were hurt during the blog post😉. I also disavow any responsibility for any hurt (real or imagined) to any past, current and future mascots. The attempt should not be looked upon as demeaning people who are accused of false crimes, tortured and confessions eked out of them as this happens quite a lot (In India for sure, but guess it’s the same world over in various degrees). The idea is loosely inspired by Chocolate:Deep Dark Secrets. (2005)

On a more positive note, let’s start –

Being a Sunday morning woke up late to find incessant knocking on the door, incidentally mum was not at home. Opening the door, found two official looking gentleman. They asked my name, asked my credentials, tortured and arrested me for “Group conspiracy of Malicious Mischief in second and third degrees” .

The torture was done by means of making me forcefully watch endless reruns of ‘Norbit‘ . While I do love Eddie Murphy, this was one of his movies he could have done without😦. I guess for many people watching it once was torture enough. I *think* they were nominated for razzie awards dunno if they won it or not, but this is beside the point.

Unlike the 20 years it takes for a typical case to reach to its conclusion even in the smallest court in India, due to the torture, I was made to confess (due to endless torture) and was given summary judgement. The judgement was/is as follows –

a. Do 100 hours of Community service in Debian in 2017. This could be done via blog posts, raising tickets in the Debian BTS or in whichever way I could be helpful to Debian.

b. Write a confessional with some photographic evidence sharing/detailing some of the other members who were part of the conspiracy in view of the reduced sentence.

So now, have been forced to write this confession –

As you all know, I won a bursary this year for debconf16. What is not known by most people is that I also got an innocuous looking e-mail titled ‘ Pollito for DPL ‘. While I can’t name all the names as investigation is still ongoing about how far-reaching the conspiracy is . The email was purportedly written by members of ‘cabal within cabal’ which are in Debian. I looked at the email header to see if this was genuine and I could trace the origin but was left none the wiser, as obviously these people are far more technically advanced than to fall in simple tricks like this –

Anyways, secretly happy that I have been invited to be part of these elites, I did the visa thing, packed my bags and came to Debconf16.

At this point in juncture, I had no idea whether it was real or I had imagined the whole thing. Then to my surprise saw this –

evidence of conspiracy to have Pollito as DPL, Wifi Password

Just like the Illuminati the conspiracy was for all to see those who knew about it. Most people were thinking of it as a joke, but those like me who had got e-mails knew better. I knew that the thing is real, now I only needed to bide my time and knew that the opportunity would present itself.

And few days later, sure enough, there was a trip planned for ‘Table Mountain, Cape Town’ . Few people planned to hike to the mountain, while few chose to take the cable car till up the mountain.

First glance of the cable car with table mountain as background

Quite a few people came along with us and bought tickets for the to and fro to the mountain and back.

Ticket for CPT Table mountain car cable

Incidentally, I was thinking if the South African Govt. were getting the tax or not. If you look at the ticket, there is just a bar-code. In India as well as the U.S. there is TIN – Tax Identification Number –

TIN displayed on an invoice from channeltimes.com

Few links to share what it is all about . While these should be on all invoices, need to specially check when taking high-value items. In India as shared in the article the awareness, knowledge leaves a bit to be desired. While I’m drifting from the incident, it would be nice if somebody from SA could share how things work there.

Moving on, we boarded the cable car. It was quite spacious cable car with I guess around 30-40 people or some more who were able to see everything along with the controller.

from inside the table mountain cable car 360 degrees

It was a pleasant cacophony of almost two dozen or more nationalities on this 360 degrees moving chamber. I was a little worried though as it essentially is a bucket and there is always a possibility that a severe wind could damage it. Later somebody did share that some frightful incidents had occurred not too long ago on the cable car.

It took about 20-25 odd minutes to get to the top of table mountain and we were presented with views such as below –

View from Table Mountain cable car looking down

The picture I am sharing is actually when we were going down as all the pictures of going up via the cable car were over-exposed. Also, it was pretty crowded on the way up then on the way down so handling the mobile camera was not so comfortable.

Once we reached up, the wind was blowing at incredible speeds. Even with my jacket and everything I was feeling cold. Most of the group around 10-12 people looked around if we could find a place to have some refreshments and get some of the energy in the body. So we all ventured to a place and placed our orders –

the bleh... Irish coffee at top of Table Mountain

I was introduced to Irish Coffee few years back and have had some incredible Irish Coffees in Pune and elsewhere. I do hope to be able to make Irish Coffee at home if and when I have my own house. This is hotter than brandy and is perfect if you are suffering from cold etc if done right, really needs some skills. This is the only drink which I wanted in SA which I never got right😦 . As South Africa was freezing for me, this would have been the perfect antidote but the one there as well as elsewhere were all …bleh.

What was interesting though, was the coffee caller besides it. It looked like a simple circuit mounted on a PCB board with lights, vibrations and RFID and it worked exactly like that. I am guessing as and when the order is ready, there is an interrupt signal sent via radio waves which causes the buzzer to light and vibrate. Here’s the back panel if somebody wants to take inspiration and try it as a fun project –

backpanel of the buzz caller

Once we were somewhat strengthened by the snacks, chai, coffee etc. we made our move to seeing the mountain. The only way to describe it is that it’s similar to Raigad Fort but the plateau seemed to be bigger. The wikipedia page of Table Mountain attempts to share but I guess it’s more clearly envisioned by one of the pictures shared therein.

table mountain panaromic image

I have to say while Table Mountain is beautiful and haunting as it has scenes like these –

Some of the oldest rocks known to wo/man.

There is something there which pulls you, which reminds you of a long lost past. I could have simply sat there for hours together but as was part of the group had to keep with them. Not that I minded.

The moment I was watching this, I was transported to some memories of the Himalayas about 20 odd years or so. In that previous life, I had the opportunity to be with some of the most beautiful women and also been in the most happening places, the Himalayas. I had shared years before some of my experiences I had in the Himalayas. I discontinued it as I didn’t have a decent camera at that point in time. While I don’t wanna digress, I would challenge anybody to experience the Himalayas and then compare. It is just something inexplicable. The beauty and the rawness that Himalayas shows makes you feel insignificant and yet part of the whole cosmos. What Paulo Cohello expressed in The Valkyries is something that could be felt in the Himalayas. Leh, Ladakh, Himachal , Garwhal, Kumaon. The list will go on forever as there are so many places, each more beautiful than the other. Most places are also extremely backpacker-friendly so if you ask around you can get some awesome deals if you want to spend more than a few days in one place.

Moving on, while making small talk @olasd or Nicolas Dandrimont , the headmaster of our trip made small talk to each of us and eked out from all of us that we wanted to have Pollito as our DPL (Debian Project Leader) for 2017. Few pictures being shared below as supporting evidence as well –

The Pollito as DPL cabal in action

members of the Pollito as DPL

where am I or more precisely how far am I from India.

While I do not know who further up than Nicolas was on the coup which would take place. The idea was this –

If the current DPL steps down, we would take all and any necessary actions to make Pollito our DPL.

Pollito going to SA - photo taken by Jonathan Carter This has been taken from Pollito’s adventure

Being a responsible journalist, I also enquired about Pollito’s true history as it would not have been complete without one. This is the e-mail I got from Gunnar Wolf, a friend and DD from Mexico🙂

Turns out, Valessio has just spent a week staying at my house🙂 And
in any case, if somebody in Debian knows about Pollito’s
childhood… That is me.

Pollito came to our lives when we went to Congreso Internacional de
Software Libre (CISOL) in Zacatecas city. I was strolling around the
very beautiful city with my wife Regina and our friend Alejandro
Miranda, and at a shop at either Ramón López Velarde or Vicente
Guerrero, we found a flock of pollitos.

http://www.openstreetmap.org/#map=17/22.77111/-102.57145

Even if this was comparable to a slave market, we bought one from
them, and adopted it as our own.

Back then, we were a young couple… Well, we were not that young
anymore. I mean, we didn’t have children. Anyway, we took Pollito with
us on several road trips, such as the only time I have crossed an
international border driving: We went to Encuentro Centroamericano de
Software Libre at Guatemala city in 2012 (again with Alejandro), and
you can see several Pollito pics at:

http://gwolf.org/album/road-trip-ecsl-2012-guatemala-0

Pollito likes travelling. Of course, when we were to Nicaragua for
DebConf, Pollito tagged along. It was his first flight as a passenger
(we never asked about his previous life in slavery; remember, Pollito
trust no one).

Pollito felt much welcome with the DebConf crowd. Of course, as
Pollito is a free spirit, we never even thought about forcing him to
come back with us. Pollito went to Switzerland, and we agreed to meet
again every year or two. It’s always nice to have a chat with him.

Hugs!

So with that backdrop I would urge fellow Debianities to take up the slogans –

LONG LIVE THE DPL !

LONG LIVE POLLITO !

LONG LIVE POLLITO THE DPL !

The first step to make Pollito the DPL is to ensure he has a @debian.org (pollito@debian.org)

We also need him to be made a DD because only then can he become a DPL.

In solidarity and in peace🙂


Filed under: Miscellenous Tagged: #caller, #confession, #Debconf16, #debian, #Fiction, #history, #Pollito, #Pollito as DPL, #Table Mountain, Cabal, memories, south africa

05 December, 2016 05:01PM by shirishag75

hackergotchi for Norbert Preining

Norbert Preining

Debian/TeX Live 2016.20161130-1

As we are moving closer to the Debian release freeze, I am shipping out a new set of packages. Nothing spectacular here, just the regular updates and a security fix that was only reported internally. Add sugar and a few minor bug fixes.
texlive2016-debian

I have been silent for quite some time, busy at my new job, busy with my little monster, writing papers, caring for visitors, living. I have quite a lot of things I want to write, but not enough time, so very short only this one.

Enjoy.

New packages

awesomebox, baskervillef, forest-quickstart, gofonts, iscram, karnaugh-map, tikz-optics, tikzpeople, unicode-bidi.

Updated packages

acmart, algorithms, aomart, apa, apa6, appendix, apxproof, arabluatex, asymptote, background, bangorexam, beamer, beebe, biblatex-gb7714-2015, biblatex-mla, biblatex-morenames, bibtexperllibs, bidi, bookcover, bxjalipsum, bxjscls, c90, cals, cell, cm, cmap, cmextra, context, cooking-units, ctex, cyrillic, dirtree, ekaia, enotez, errata, euler, exercises, fira, fonts-churchslavonic, formation-latex-ul, german, glossaries, graphics, handout, hustthesis, hyphen-base, ipaex, japanese, jfontmaps, kpathsea, l3build, l3experimental, l3kernel, l3packages, latex2e-help-texinfo-fr, layouts, listofitems, lshort-german, manfnt, mathastext, mcf2graph, media9, mflogo, ms, multirow, newpx, newtx, nlctdoc, notes, patch, pdfscreen, phonenumbers, platex, ptex, quran, readarray, reledmac, shapes, showexpl, siunitx, talk, tcolorbox, tetex, tex4ht, texlive-en, texlive-scripts, texworks, tikz-dependency, toptesi, tpslifonts, tracklang, tugboat, tugboat-plain, units, updmap-map, uplatex, uspace, wadalab, xecjk, xellipsis, xepersian, xint.

05 December, 2016 02:58PM by Norbert Preining

Reproducible builds folks

Reproducible Builds: week 84 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 27 and Saturday December 3 2016:

Reproducible work in other projects

Media coverage, etc.

  • There was a Reproducible Builds hackathon in Boston with contributions from Dafydd, Valerie, Clint, Harlen, Anders, Robbie and Ben. (See the "Bugs filed" section below for the results).

  • Distrowatch mentioned Webconverger's reproducible status.

Bugs filed

Chris Lamb:

Clint Adams:

Dafydd Harries:

Daniel Shahaf:

Reiner Herrmann:

Valerie R Young:

Reviews of unreproducible packages

15 package reviews have been added, 4 have been updated and 26 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been added:

Weekly QA work

During our reproducibility testing, some FTBFS bugs have been detected and reported by:

  • Chris Lamb (5)
  • Lucas Nussbaum (8)
  • Santiago Vila (1)

diffoscope development

Is is available now in Debian, Archlinux and on PyPI.

strip-nondeterminism development

  • At the Reproducible Builds Boston hackathon Anders Kaseorg filed #846895 treat .par files as Zip archives, including a patch which was merged into master.

reprotest development

tests.reproducible-builds.org

  • Holger made a couple of changes:

    • Group all "done" and all "open" usertagged bugs together in the bugs graphs and move the "done bugs" from the bottom of these gaps.
    • Update list of packages installed on .debian.org machines.
    • Made the maintenance jobs run every 2h instead of 3h.
    • Various bug fixes and minor improvements.
  • After thorough review by Mattia, some patches by Valerie were merged in preparation of the switch from sqlite to Postgresql, most notably a conversion to the sqlalchemy expression language.

  • Holger gave a talk at Profitbricks about how Debian is using 168 cores, 503 GB RAM and 5 TB storage to make jenkins.debian.net and tests.reproducible-builds.org run. Many thanks to Profitbricks for supporting jenkins.debian.net since August 2012!

  • Holger created a Jenkins job to build reprotest from git master branch.

  • Finally, the Jenkins Naginator plugin was installed to retry git cloning in case of Alioth/network failures, this will benefit all jobs using Git on jenkins.debian.net.

Misc.

This week's edition was written by Chris Lamb, Valerie Young, Vagrant Cascadian, Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

05 December, 2016 12:31PM

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in November 2016

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Android

  • Chris Lamb was so kind to send in a patch for apktool to make the build reproducible (#845475). Although this was not enough to fix the issue it set me on the right path to eventually resolve bug number 845475.

Debian Games

  • I packaged a couple of new upstream releases for extremetuxracer, fifechan, fife, unknown-horizons, freeciv, atanks and armagetronad. Most notably fifechan was accepted by the FTP team which allowed me to package new versions of fife and unknown-horizons which are both back in testing again. I expect that upstream will make their final release sometime in December. Atanks has been orphaned a while ago and since upstream is still active and I kinda like the game I decided to adopt it. I also uploaded a backport of Freeciv 2.5.6 to jessie-backports.
  • In November we received a bunch of RC bug reports again because, hey, it is almost time for the Freeze, let’s break some packages. Thus I spent some time fixing freeorion (#843132), pokerth (#843078), simutrans (#828545), freeciv (#844198) and warzone2100 (#844870).
  • I also updated the debian-games blend, we are at version 1.6 now, and made some smaller adjustments. The most important change was adding a new binary package, games-all, that installs..well, all! I know this will make at least one person on this planet happy. Actually I was kind of forced into adding it because blends-dev automatically creates it as a requirement for choosing blends with the Debian Installer. But don’t be afraid games-all only recommends games-finest, the rest is suggested.
  • Last but not least I worked on performous and could close a wishlist bug report (#425898). The submitter asked to suggest some free song packages for this karaoke game.

Debian Java

  • I sponsored uncommons-watchmaker for Kai-Chung and also reviewed libnative-platform-java and granted upload rights to him.
  • I packaged new upstream releases of lombok-patcher, electric, undertow, sweethome3d and sweethome3d-furniture-editor.
  • I spent quite some time on reviewing (especially the copyright review took most of the time) and improving the packaging for tycho (#816604) which is a precondition for packaging the latest upstream release of Eclipse, a popular Java IDE. Luca Vercelli has been working on it for the last couple of months and he did most of the initial packaging. Unfortunately I was only able to upload the package last week which means that the chances for updating Eclipse for Stretch are slim.
  • Due to time constraints I could not finish the Netbeans update in time which I had started back in October. This is on my priority list for December now.
  • Several security issues were reported against Tomcat{6,7,8}. I helped with reviewing some of the patches that Emmanuel prepared for Jessie and worked on fixing the same bugs in Wheezy.

Debian LTS

This was my ninth month as a paid contributor and I have been paid to work 11 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 14. November until 21. November I was in charge of our LTS frontdesk. I triaged bugs in teeworlds, libdbd-mysql-perl, bash, libxml2, tiff, firefox-esr, drupal7, moin, libgc, w3m and sniffit.
  • DLA-715-1. Issued a security update for drupal7 fixing 2 CVE.
  • DLA-717-1. Issued a security update for moin fixing 2 CVE.
  • DLA-728-1. Issued a security update for tomcat6 fixing 8 CVE. (Debian bug #845385 was assigned a CVE later).
  • DLA-729-1. Issued a security update for tomcat7 fixing 8 CVE. (Debian bug #845385 was assigned a CVE later).
  • Especially the patches and the subsequent testing for CVE-2016-0762 and CVE-2016-6816 required most of the time.

Non-maintainer uploads

  • I uploaded an NMU for angband to fix #837394. The patch was kindly prepared by Adrian Bunk.

It is already this time of the year again. See you next year for another report. 🙂

05 December, 2016 12:48AM by Apo

hackergotchi for Ben Hutchings

Ben Hutchings

Linux Kernel Summit 2016, part 2

I attended this year's Linux Kernel Summit in Santa Fe, NM, USA and made notes on some of the sessions that were relevant to Debian. LWN also reported many of the discussions. This is the second and last part of my notes; part 1 is here.

Updated: I corrected the description of which Intel processors support SMEP.

Kernel Hardening

Kees Cook presented the ongoing work on upstream kernel hardening, also known as the Kernel Self-Protection Project or KSPP.

GCC plugins

The kernel build system can now build and use GCC plugins to implement some protections. This requires gcc 4.5 and the plugin headers installed. It has been tested on x86, arm, and arm64. It is disabled by CONFIG_COMPILE_TEST because CI systems using allmodconfig/allyesconfig probably don't have those installed, but this ought to be changed at some point.

There was a question as to how plugin headers should be installed for cross-compilers or custom compilers, but I didn't hear a clear answer to this. Kees has been prodding distribution gcc maintainers to package them. Mark Brown mentioned the Linaro toolchain being widely used; Kees has not talked to its maintainers yet.

Probabilistic protections

These protections are based on hidden state that an attacker will need to discover in order to make an effective attack; they reduce the probability of success but don't prevent it entirely.

Kernel address space layout randomisation (KASLR) has now been implemented on x86, arm64, and mips for the kernel image. (Debian enables this.) However there are still lots of information leaks that defeat this. This could theoretically be improved by relocating different sections or smaller parts of the kernel independently, but this requires re-linking at boot. Aside from software information leaks, the branch target predictor on (common implementations of) x86 provides a side channel to find addresses of branches in the kernel.

Page and heap allocation, etc., is still quite predictable.

struct randomisation (RANDSTRUCT plugin from grsecurity) reorders members in (a) structures containing only function pointers (b) explicitly marked structures. This makes it very hard to attack custom kernels where the kernel image is not readable. But even for distribution kernels, it increases the maintenance burden for attackers.

Deterministic protections

These protections block a class of attacks completely.

Read-only protection of kernel memory is either mandatory or enabled by default on x86, arm, and arm64. (Debian enables this.)

Protections against execution of user memory in kernel mode are now implemented in hardware on x86 (SMEP, in Intel processors from Skylake Broadwell onward) and on arm64 (PXN, from ARMv8.1). But Skylake Broadwell is not available for servers in high-end server variants and ARMv8.1 is not yet implemented at all! s390 always had this protection.

It may be possible to 'emulate' this using other hardware protections. arm (v7) and arm64 now have this, but x86 doesn't. Linus doesn't like the overhead of previously proposed implementations for x86. It is possible to do this using PCID (in Intel processors from Sandy Bridge onward), which has already been done in PaX - and this should be fast enough.

Virtually mapped stacks protect against stack overflow attacks. They were implemented as an option for x86 only in 4.9. (Debian enables this.)

Copies to or from user memory sometimes use a user-controlled size that is not properly bounded. Hardened usercopy, implemented as an option in 4.8 for many architectures, protects against this. (Debian enables this.)

Memory wiping (zero on free) protects against some information leaks and use-after-free bugs. It was already implemented as debug feature with non-zero poison value, but at some performance cost. Zeroing can be cheaper since it allows allocator to skip zeroing on reallocation. That was implemented as an option in 4.6. (Debian does not currently enable this but we might do if the performance cost is low enough.)

Constification (with the CONSTIFY gcc plugin) reduces the amount of static data that can be written to. As with RANDSTRUCT, this is applied to function pointer tables and explicitly marked structures. Instances of some types need to be modified very occasionally. In PaX/Grsecurity this is done with pax_{open,close}_kernel() which globally disable write protection temporarily. It would be preferable to override write protection in a more directed way, so that the permission to write doesn't leak into any other code that interrupts this process. The feature is not in mainline yet.

Atomic wrap detction protects against reference-counting bugs which can result in a use-after-free. Overflow and underflow are trapped and result in an 'oops'. There is no measurable performance impact. It would be applied to all operations on the atomic_t type, but there needs to be an opt-out for atomics that are not ref-counters - probably by adding an atomic_wrap_t type for them. This has been implemented for x86, arm, and arm64 but is not in mainline yet.

Kernel Freezer Hell

For the second year running, Jiri Kosina raised the problem of 'freezing' kthreads (kernel-mode threads) in preparation for system suspend (suspend to RAM, or hibernation). What are the semantics? What invariants should be met when a kthread gets frozen? They are not defined anywhere.

Most freezable threads don't actually need to be quiesced. Also many non-freezable threads are pointlessly calling try_to_freeze() (probably due to copying code without understanding it)).

At a system level, what we actually need is I/O and filesystem consistency. This should be achieved by:

  • Telling mounted filesystems to freeze. They can quiesce any kthreads they created.
  • Device drivers quiescing any kthreads they created, from their PM suspend implementation.

The system suspend code should not need to directly freeze threads.

Kernel Documentation

Jon Corbet and Mauro Carvalho presented the recent work on kernel documentation.

The kernel's documentation system was a house of cards involving DocBook and a lot of custom scripting. Both the DocBook templates and plain text files are gradually being converted to reStructuredText format, processed by Sphinx. However, manual page generation is currently 'broken' for documents processed by Sphinx.

There are about 150 files at the top level of the documentation tree, that are being gradually moved into subdirectories. The most popular files, that are likely to be referenced in external documentation, have been replaced by placeholders.

Sphinx is highly extensible and this has been used to integrate kernel-doc. It would be possible to add extensions that parse and include the MAINTAINERS file and Documentation/ABI/ files, which have their own formats, but the documentation maintainers would prefer not to add extensions that can't be pushed to Sphinx upstream.

There is lots of obsolete documentation, and patches to remove those would be welcome.

Linus objected to PDF files recently added under the Documentation/media directory - they are not the source format so should not be there! They should be generated from the corresponding SVG or image files at build time.

Issues around Tracepoints

Steve Rostedt and Shuah Khan led a discussion about tracepoints. Currently each maintainer decides which tracepoints to create. The cost of each added tracepoint is minimal, but the cost of very many tracepoints is more substantial. So there is such a thing as too many tracepoints, and we need a policy to decide when they are justified. They advised not to create tracepoints just in case, since kprobes can be used for tracing (almost) anywhere dynamically.

There was some support for requiring documentation of each new tracepoint. That may dissuade introduction of obscure tracepoints, but also creates a higher expectation of stability.

Tools such as bcc and IOVisor are now being created that depend on specific tracepoints or even function names (through kprobes). Should we care about breaking them?

Linus said that we should strive to be polite to developers and users relying on tracepoints, but if it's too painful to maintain a tracepoint then we should go ahead and change it. Where the end users of the tool are themselves developers it's more reasonable to expect them to upgrade the tool and we should care less about changing it. In some cases tracepoints could provide dummy data for compatibility (as is done in some places in procfs).

05 December, 2016 12:01AM

December 04, 2016

Linux Kernel Summit 2016, part 2

I attended this year's Linux Kernel Summit in Santa Fe, NM, USA and made notes on some of the sessions that were relevant to Debian. LWN also reported many of the discussions. This is the second and last part of my notes; part 1 is here.

Kernel Hardening

Kees Cook presented the ongoing work on upstream kernel hardening, also known as the Kernel Self-Protection Project or KSPP.

GCC plugins

The kernel build system can now build and use GCC plugins to implement some protections. This requires gcc 4.5 and the plugin headers installed. It has been tested on x86, arm, and arm64. It is disabled by CONFIG_COMPILE_TEST because CI systems using allmodconfig/allyesconfig probably don't have those installed, but this ought to be changed at some point.

There was a question as to how plugin headers should be installed for cross-compilers or custom compilers, but I didn't hear a clear answer to this. Kees has been prodding distribution gcc maintainers to package them. Mark Brown mentioned the Linaro toolchain being widely used; Kees has not talked to its maintainers yet.

Probabilistic protections

These protections are based on hidden state that an attacker will need to discover in order to make an effective attack; they reduce the probability of success but don't prevent it entirely.

Kernel address space layout randomisation (KASLR) has now been implemented on x86, arm64, and mips for the kernel image. (Debian enables this.) However there are still lots of information leaks that defeat this. This could theoretically be improved by relocating different sections or smaller parts of the kernel independently, but this requires re-linking at boot. Aside from software information leaks, the branch target predictor on (common implementations of) x86 provides a side channel to find addresses of branches in the kernel.

Page and heap allocation, etc., is still quite predictable.

struct randomisation (RANDSTRUCT plugin from grsecurity) reorders members in (a) structures containing only function pointers (b) explicitly marked structures. This makes it very hard to attack custom kernels where the kernel image is not readable. But even for distribution kernels, it increases the maintenance burden for attackers.

Deterministic protections

These protections block a class of attacks completely.

Read-only protection of kernel memory is either mandatory or enabled by default on x86, arm, and arm64. (Debian enables this.)

Protections against execution of user memory in kernel mode are now implemented in hardware on x86 (SMEP, in Intel processors from Skylake onward) and on arm64 (PXN, from ARMv8.1). But Skylake is not available for servers and ARMv8.1 is not yet implemented at all! s390 always had this protection.

It may be possible to 'emulate' this using other hardware protections. arm (v7) and arm64 now have this, but x86 doesn't. Linus doesn't like the overhead of previously proposed implementations for x86. It is possible to do this using PCID (in Intel processors from Sandy Bridge onward), which has already been done in PaX - and this should be fast enough.

Virtually mapped stacks protect against stack overflow attacks. They were implemented as an option for x86 only in 4.9. (Debian enables this.)

Copies to or from user memory sometimes use a user-controlled size that is not properly bounded. Hardened usercopy, implemented as an option in 4.8 for many architectures, protects against this. (Debian enables this.)

Memory wiping (zero on free) protects against some information leaks and use-after-free bugs. It was already implemented as debug feature with non-zero poison value, but at some performance cost. Zeroing can be cheaper since it allows allocator to skip zeroing on reallocation. That was implemented as an option in 4.6. (Debian does not currently enable this but we might do if the performance cost is low enough.)

Constification (with the CONSTIFY gcc plugin) reduces the amount of static data that can be written to. As with RANDSTRUCT, this is applied to function pointer tables and explicitly marked structures. Instances of some types need to be modified very occasionally. In PaX/Grsecurity this is done with pax_{open,close}_kernel() which globally disable write protection temporarily. It would be preferable to override write protection in a more directed way, so that the permission to write doesn't leak into any other code that interrupts this process. The feature is not in mainline yet.

Atomic wrap detction protects against reference-counting bugs which can result in a use-after-free. Overflow and underflow are trapped and result in an 'oops'. There is no measurable performance impact. It would be applied to all operations on the atomic_t type, but there needs to be an opt-out for atomics that are not ref-counters - probably by adding an atomic_wrap_t type for them. This has been implemented for x86, arm, and arm64 but is not in mainline yet.

Kernel Freezer Hell

For the second year running, Jiri Kosina raised the problem of 'freezing' kthreads (kernel-mode threads) in preparation for system suspend (suspend to RAM, or hibernation). What are the semantics? What invariants should be met when a kthread gets frozen? They are not defined anywhere.

Most freezable threads don't actually need to be quiesced. Also many non-freezable threads are pointlessly calling try_to_freeze() (probably due to copying code without understanding it)).

At a system level, what we actually need is I/O and filesystem consistency. This should be achieved by:

  • Telling mounted filesystems to freeze. They can quiesce any kthreads they created.
  • Device drivers quiescing any kthreads they created, from their PM suspend implementation.

The system suspend code should not need to directly freeze threads.

Kernel Documentation

Jon Corbet and Mauro Carvalho presented the recent work on kernel documentation.

The kernel's documentation system was a house of cards involving DocBook and a lot of custom scripting. Both the DocBook templates and plain text files are gradually being converted to reStructuredText format, processed by Sphinx. However, manual page generation is currently 'broken' for documents processed by Sphinx.

There are about 150 files at the top level of the documentation tree, that are being gradually moved into subdirectories. The most popular files, that are likely to be referenced in external documentation, have been replaced by placeholders.

Sphinx is highly extensible and this has been used to integrate kernel-doc. It would be possible to add extensions that parse and include the MAINTAINERS file and Documentation/ABI/ files, which have their own formats, but the documentation maintainers would prefer not to add extensions that can't be pushed to Sphinx upstream.

There is lots of obsolete documentation, and patches to remove those would be welcome.

Linus objected to PDF files recently added under the Documentation/media directory - they are not the source format so should not be there! They should be generated from the corresponding SVG or image files at build time.

Issues around Tracepoints

Steve Rostedt and Shuah Khan led a discussion about tracepoints. Currently each maintainer decides which tracepoints to create. The cost of each added tracepoint is minimal, but the cost of very many tracepoints is more substantial. So there is such a thing as too many tracepoints, and we need a policy to decide when they are justified. They advised not to create tracepoints just in case, since kprobes can be used for tracing (almost) anywhere dynamically.

There was some support for requiring documentation of each new tracepoint. That may dissuade introduction of obscure tracepoints, but also creates a higher expectation of stability.

Tools such as bcc and IOVisor are now being created that depend on specific tracepoints or even function names (through kprobes). Should we care about breaking them?

Linus said that we should strive to be polite to developers and users relying on tracepoints, but if it's too painful to maintain a tracepoint then we should go ahead and change it. Where the end users of the tool are themselves developers it's more reasonable to expect them to upgrade the tool and we should care less about changing it. In some cases tracepoints could provide dummy data for compatibility (as is done in some places in procfs).

04 December, 2016 09:18PM

Niels Thykier

Piuparts integration in britney

As of today, britney now fetches reports from piuparts.debian.org and uses it as a part of her evaluation for package migration.  As with her RC bug check, we are only preventing (known) regressions from migrating.

The messages (subject to change) look something like:

  • Piuparts tested OK
  • Rejected due to piuparts regression
  • Ignoring piuparts failure (Not a regression)
  • Cannot be tested by piuparts (not a blocker)

If you want to do machine parsing of the Britney excuses, we also provide an excuses.yaml. In there, you are looking for “excuses[X].policy_info.piuparts.test-results”, which will be one of:

  • pass
  • regression
  • failed
  • cannot-be-tested

Enjoy.🙂

 


Filed under: Debian, Release-Team

04 December, 2016 11:06AM by Niels Thykier

hackergotchi for Jo Shields

Jo Shields

A quick introduction to Flatpak

Releasing ISV applications on Linux is often hard. The ABI of all the libraries you need changes seemingly weekly. Hence you have the option of bundling the world, or building a thousand releases to cover a thousand distribution versions. As a case in point, when MonoDevelop started bundling a C Git library instead of using a C# git implementation, it gained dependencies on all sorts of fairly weak ABI libraries whose exact ABI mix was not consistent across any given pair of distro releases. This broke our policy of releasing “works on anything” .deb and .rpm packages. As a result, I pretty much gave up on packaging MonoDevelop upstream with version 5.10.

Around the 6.1 release window, I decided to take re-evaluate question. I took a closer look at some of the fancy-pants new distribution methods that get a lot of coverage in the Linux press: Snap, AppImage, and Flatpak.

I started with AppImage. It’s very good and appealing for its specialist areas (no external requirements for end users), but it’s kinda useless at solving some of our big areas (the ABI-vs-bundling problem, updating in general).

Next, I looked at Flatpak (once xdg-app). I liked the concept a whole lot. There’s a simple 3-tier dependency hierarchy: Applications, Runtimes, and Extensions. An application depends on exactly one runtime.  Runtimes are root-level images with no dependencies of their own. Extensions are optional add-ons for applications. Anything not provided in your target runtime, you bundle. And an integrated updates mechanism allows for multiple branches and multiple releases parallel-installed (e.g. alpha & stable, easily switched).

There’s also security-related sandboxing features, but my main concerns on a first examination were with the dependency and distribution questions. That said, some users might be happier running Microsoft software on their Linux desktop if that software is locked up inside a sandbox, so I’ve decided to embrace that functionality rather than seek to avoid it.

I basically stopped looking at this point (sorry Snap!). Flatpak provided me with all the functionality I wanted, with an extremely helpful and responsive upstream. I got to work on trying to package up MonoDevelop.

Flatpak (optionally!) uses a JSON manifest for building stuff. Because Mono is still largely stuck in a Gtk+2 world, I opted for the simplest runtime, org.freedesktop.Runtime, and bundled stuff like Gtk+ into the application itself.

Some gentle patching here & there resulted in this repository. Every time I came up with an exciting new edge case, upstream would suggest a workaround within hours – or failing that, added new features to Flatpak just to support my needs (e.g. allowing /dev/kvm to optionally pass through the sandbox).

The end result is, as of the upcoming 0.8.0 release of Flatpak, from a clean install of the flatpak package to having a working MonoDevelop is a single command: flatpak install --user --from https://download.mono-project.com/repo/monodevelop.flatpakref 

For the current 0.6.x versions of Flatpak, the user also needs to flatpak remote-add --user --from gnome https://sdk.gnome.org/gnome.flatpakrepo first – this step will be automated in 0.8.0. This will download org.freedesktop.Runtime, then com.xamarin.MonoDevelop; export icons ‘n’ stuff into your user environment so you can just click to start.

There’s some lingering experience issues due the sandbox which are on my radar. “Run on external console” doesn’t work, for example, or “open containing folder”. There are people working on that (a missing DBus# feature to allow breaking out of the sandbox). But overall, I’m pretty happy. I won’t be entirely satisfied until I have something approximating feature equivalence to the old .debs.  I don’t think that will ever quite be there, since there’s just no rational way to allow arbitrary /usr stuff into the sandbox, but it should provide a decent basis for a QA-able, supportable Linux MonoDevelop. And we can use this work as a starting point for any further fancy features on Linux.

Gtk# app development in Flatpak MonoDevelop

Editing MonoDevelop in MonoDevelop. *Inception noise*

04 December, 2016 10:44AM by directhex

December 03, 2016

hackergotchi for Ben Hutchings

Ben Hutchings

Linux Kernel Summit 2016, part 1

I attended this year's Linux Kernel Summit in Santa Fe, NM, USA and made notes on some of the sessions that were relevant to Debian. LWN also reported many of the discussions. This is the first of two parts of my notes; part 2 is here.

Stable process

Jiri Kosina, in his role as a distribution maintainer, sees too many unsuitable patches being backported - e.g. a fix for a bug that wasn't present or a change that depends on an earlier semantic change so that when cherry-picked it still compiles but isn't quite right. He thinks the current review process is insufficient to catch them.

As an example, a recent fix for a minor information leak (CVE-2016-9178) depended on an earlier change to page fault handling. When backported by itself, it introduced a much more serious security flaw (CVE-2016-9644). This could have been caught very quickly by a system call fuzzer.

Possible solutions: require 'Fixes' field, not just 'Cc: stable'. Deals with 'bug wasn't present', but not semantic changes.

There was some disagreement whether 'Fixes' without 'Cc: stable' should be sufficient for inclusion in stable. Ted Ts'o said he specifically does that in some cases where he thinks backporting is risky. Greg Kroah-Hartman said he takes it as a weaker hint for inclusion in stable.

Is it a good idea to keep 'Cc: stable' given the risk of breaking embargo? On balance, yes, it only happened once.

Sometimes it's hard to know exactly how/when the bug was introduced. Linus doesn't want people to guess and add incorrect 'Fixes' fields. There is still the option to give some explanation and hints for stable maintainers in the commit message. Ideally the upstream developer should provide a test case for the bug.

Is Linus happy?

Linus complained about minor fixes coming later in the release cycle. After rc2, all fixes should either be for new code introduced in the current release cycle or for important bugs. However, new, production-ready drivers without new infrastructure dependencies are welcome at almost any point in the release cycle.

He was unhappy about some big changes in RDMA, but I'm not sure what those were.

Bugzilla and bug tracking

Laura Abbott started a discussion of bugzilla.kernel.org, talking about subsystems where maintainers ignore it and any responses come from random people giving bad advice. This is a terrible experience for users. Several maintainers are actively opposed to using it, and the email bridge no longer works (or not well?). She no longer recommends Fedora bug submitters to submit reports there.

Are there any alternatives? None were proposed.

Someone asked whether Bugzilla could tell reporters to use email for certain products/components instead of continuing with the bug entry process.

Konstantin Ryabitsev talked about the difficulty of upgrading a customised instance of Bugzilla. Much customisation requires patches which don't apply to next version (maybe due to limitations of the extension mechanism?). He has had to drop many such patches.

Email is hard to track when a bug is handed over from one maintainer to another. Email archives are very unreliable. Linus: I'll take Bugzilla over mail-archive.

No-one is currently keeping track of bugs across the kernel and making sure they get addressed by an appropriate maintainer. It's (at least) a full-time job but no individual company has business case for paying for this. Konstantin suggested (I think) that CII might pay for this.

There was some discussion of what information should be included in a bug report. The Cut here line in oops messages was said to be a mistake because there are often relevant messages before it. The model of computer is often important. Beyond that, there was not much interest in the automated information gathering that distributions do. Distribution maintainers should curate bugs before forwarding upstream.

There was a request for custom fields per component in Bugzilla. Konstantin says this is doable (possibly after upgrade to version 5); it doesn't require patches.

The future of the Kernel Summit

The kernel community is growing, and the invitation list for the core day is too small to include all the right people for technical subjects. For 2017, the core half-day will have an even smaller invitation list, only ~30 subsystem maintainers that Linus pulls from. The entire technical track will be open (I think).

Kernel Summit 2017 and some mini-summits will be held in Prague alongside Open Source Summit Europe (formerly LinuxCon Europe) and Embedded Linux Conference Europe. There were some complaints that LinuxCon is not that interesting to kernel developers, compared to Linux Plumbers Conference (which followed this year's Kernel Summit). However, the Linux Foundation is apparently soliciting more hardcore technical sessions.

Kernel Summit and Linux Plumbers Conference are quite small, and it's apparently hard to find venues for them in cities that also have major airports. It might be more practical to co-locate them both with Open Source Summit in future.

time_t and 2038

On 32-bit architectures the kernel's representation of real time (time_t etc.) will break in early 2038. Fixing this in a backward-compatible way is a complex problem.

Arnd Bergmann presented the current status of this process. There has not yet been much progress in mainline, but more fixes have been prepared. The changes to struct inode and to input events are proving to be particularly problematic. There is a need to add new system calls, and he intends to add these for all (32-bit) achitectures at once.

Copyright retention

James Bottomley talked about how developers can retain copyright on their contributions. It's hard to renegotiate within an existing employment; much easier to do this when preparing to sign a new contract.

Some employers expect you to fill in a document disclosing 'prior inventions' you have worked on. Depending on how it's worded, this may require the employer to negotiate with you again whenever they want you to work on that same software.

It's much easier for contractors to retain copyright on their work - customers expect to have a custom agreement and don't expect to get copyright on contractor's software.

03 December, 2016 11:54PM

hackergotchi for Vincent Bernat

Vincent Bernat

Build-time dependency patching for Android

This post shows how to patch an external dependency for an Android project at build-time with Gradle. This leverages the Transform API and Javassist, a Java bytecode manipulation tool.

buildscript {
    dependencies {
        classpath 'com.android.tools.build:gradle:2.2.+'
        classpath 'com.android.tools.build:transform-api:1.5.+'
        classpath 'org.javassist:javassist:3.21.+'
        classpath 'commons-io:commons-io:2.4'
    }
}

Disclaimer: I am not a seasoned Android programmer, so take this with a grain of salt.

Context§

This section adds some context to the example. Feel free to skip it.

Dashkiosk is an application to manage dashboards on many displays. It provides an Android application you can install on one of those cheap Android sticks. Under the table, the application is an embedded webview backed by the Crosswalk Project web runtime which brings an up-to-date web engine, even for older versions of Android1.

Recently, a security vulnerability has been spotted in how invalid certificates were handled. When a certificate cannot be verified, the webview defers the decision to the host application by calling the onReceivedSslError() method:

Notify the host application that an SSL error occurred while loading a resource. The host application must call either callback.onReceiveValue(true) or callback.onReceiveValue(false). Note that the decision may be retained for use in response to future SSL errors. The default behavior is to pop up a dialog.

The default behavior is specific to Crosswalk webview: the Android builtin one just cancels the load. Unfortunately, the fix applied by Crosswalk is different and, as a side effect, the onReceivedSslError() method is not invoked anymore2.

Dashkiosk comes with an option to ignore TLS errors3. The mentioned security fix breaks this feature. The following example will demonstrate how to patch Crosswalk to recover the previous behavior4.

Simple method replacement§

Let’s replace the shouldDenyRequest() method from the org.xwalk.core.internal.SslUtil class with this version:

// In SslUtil class
public static boolean shouldDenyRequest(int error) {
    return false;
}

Transform registration§

Gradle Transform API enables the manipulation of compiled class files before they are converted to DEX files. To declare a transform and register it, include the following code in your build.gradle:

import com.android.build.api.transform.Context
import com.android.build.api.transform.QualifiedContent
import com.android.build.api.transform.Transform
import com.android.build.api.transform.TransformException
import com.android.build.api.transform.TransformInput
import com.android.build.api.transform.TransformOutputProvider
import org.gradle.api.logging.Logger

class PatchXWalkTransform extends Transform {
    Logger logger = null;

    public PatchXWalkTransform(Logger logger) {
        this.logger = logger
    }

    @Override
    String getName() {
        return "PatchXWalk"
    }

    @Override
    Set<QualifiedContent.ContentType> getInputTypes() {
        return Collections.singleton(QualifiedContent.DefaultContentType.CLASSES)
    }

    @Override
    Set<QualifiedContent.Scope> getScopes() {
        return Collections.singleton(QualifiedContent.Scope.EXTERNAL_LIBRARIES)
    }

    @Override
    boolean isIncremental() {
        return true
    }

    @Override
    void transform(Context context,
                   Collection<TransformInput> inputs,
                   Collection<TransformInput> referencedInputs,
                   TransformOutputProvider outputProvider,
                   boolean isIncremental) throws IOException, TransformException, InterruptedException {
        // We should do something here
    }
}

// Register the transform
android.registerTransform(new PatchXWalkTransform(logger))

The getInputTypes() method should return the set of types of data consumed by the transform. In our case, we want to transform classes. Another possibility is to transform resources.

The getScopes() method should return a set of scopes for the transform. In our case, we are only interested by the external libraries. It’s also possible to transform our own classes.

The isIncremental() method returns true because we support incremental builds.

The transform() method is expected to take all the provided inputs and copy them (with or without modifications) to the location supplied by the output provider. We didn’t implement this method yet. This causes the removal of all external dependencies from the application.

Noop transform§

To keep all external dependencies unmodified, we must copy them:

@Override
void transform(Context context,
               Collection<TransformInput> inputs,
               Collection<TransformInput> referencedInputs,
               TransformOutputProvider outputProvider,
               boolean isIncremental) throws IOException, TransformException, InterruptedException {
    inputs.each {
        it.jarInputs.each {
            def jarName = it.name
            def src = it.getFile()
            def dest = outputProvider.getContentLocation(jarName, 
                                                         it.contentTypes, it.scopes,
                                                         Format.JAR);
            def status = it.getStatus()
            if (status == Status.REMOVED) { // ❶
                logger.info("Remove ${src}")
                FileUtils.delete(dest)
            } else if (!isIncremental || status != Status.NOTCHANGED) { // ❷
                logger.info("Copy ${src}")
                FileUtils.copyFile(src, dest)
            }
        }
    }
}

We also need two additional imports:

import com.android.build.api.transform.Status
import org.apache.commons.io.FileUtils

Since we are handling external dependencies, we only have to manage JAR files. Therefore, we only iterate on jarInputs and not on directoryInputs. There are two cases when handling incremental build: either the file has been removed (❶) or it has been modified (❷). In all other cases, we can safely assume the file is already correctly copied.

JAR patching§

When the external dependency is the Crosswalk JAR file, we also need to modify it. Here is the first part of the code (replacing ❷):

if ("${src}" ==~ ".*/org.xwalk/xwalk_core.*/classes.jar") {
    def pool = new ClassPool()
    pool.insertClassPath("${src}")
    def ctc = pool.get('org.xwalk.core.internal.SslUtil') // ❸

    def ctm = ctc.getDeclaredMethod('shouldDenyRequest')
    ctc.removeMethod(ctm) // ❹

    ctc.addMethod(CtNewMethod.make("""
public static boolean shouldDenyRequest(int error) {
    return false;
}
""", ctc)) // ❺

    def sslUtilBytecode = ctc.toBytecode() // ❻

    // Write back the JAR file
    // …
} else {
    logger.info("Copy ${src}")
    FileUtils.copyFile(src, dest)
}

We also need the following additional imports to use Javassist:

import javassist.ClassPath
import javassist.ClassPool
import javassist.CtNewMethod

Once we have located the JAR file we want to modify, we add it to our classpath and retrieve the class we are interested in (❸). We locate the appropriate method and delete it (❹). Then, we add our custom method using the same name (❺). The whole operation is done in memory. We retrieve the bytecode of the modified class in ❻.

The remaining step is to rebuild the JAR file:

def input = new JarFile(src)
def output = new JarOutputStream(new FileOutputStream(dest))

// ❼
input.entries().each {
    if (!it.getName().equals("org/xwalk/core/internal/SslUtil.class")) {
        def s = input.getInputStream(it)
        output.putNextEntry(new JarEntry(it.getName()))
        IOUtils.copy(s, output)
        s.close()
    }
}

// ❽
output.putNextEntry(new JarEntry("org/xwalk/core/internal/SslUtil.class"))
output.write(sslUtilBytecode)

output.close()

We need the following additional imports:

import java.util.jar.JarEntry
import java.util.jar.JarFile
import java.util.jar.JarOutputStream
import org.apache.commons.io.IOUtils

There are two steps. In ❼, all classes are copied to the new JAR, except the SslUtil class. In ❽, the modified bytecode for SslUtil is added to the JAR.

That’s all! You can view the complete example on GitHub.

More complex method replacement§

In the above example, the new method doesn’t use any external dependency. Let’s suppose we also want to replace the sslErrorFromNetErrorCode() method from the same class with the following one:

import org.chromium.net.NetError;
import android.net.http.SslCertificate;
import android.net.http.SslError;

// In SslUtil class
public static SslError sslErrorFromNetErrorCode(int error,
                                                SslCertificate cert,
                                                String url) {
    switch(error) {
        case NetError.ERR_CERT_COMMON_NAME_INVALID:
            return new SslError(SslError.SSL_IDMISMATCH, cert, url);
        case NetError.ERR_CERT_DATE_INVALID:
            return new SslError(SslError.SSL_DATE_INVALID, cert, url);
        case NetError.ERR_CERT_AUTHORITY_INVALID:
            return new SslError(SslError.SSL_UNTRUSTED, cert, url);
        default:
            break;
    }
    return new SslError(SslError.SSL_INVALID, cert, url);
}

The major difference with the previous example is that we need to import some additional classes.

Android SDK import§

The classes from the Android SDK are not part of the external dependencies. They need to be imported separately. The full path of the JAR file is:

androidJar = "${android.getSdkDirectory().getAbsolutePath()}/platforms/" +
             "${android.getCompileSdkVersion()}/android.jar"

We need to load it before adding the new method into SslUtil class:

def pool = new ClassPool()
pool.insertClassPath(androidJar)
pool.insertClassPath("${src}")
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
// …

External dependency import§

We must also import org.chromium.net.NetError and therefore, we need to put the appropriate JAR in our classpath. The easiest way is to iterate through all the external dependencies and add them to the classpath.

def pool = new ClassPool()
pool.insertClassPath(androidJar)
inputs.each {
    it.jarInputs.each {
        def jarName = it.name
        def src = it.getFile()
        def status = it.getStatus()
        if (status != Status.REMOVED) {
            pool.insertClassPath("${src}")
        }
    }
}
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
pool.importPackage('org.chromium.net.NetError');
ctc.addMethod(CtNewMethod.make("…"))
// Then, rebuild the JAR...

Happy hacking!


  1. Before Android 4.4, the webview was severely outdated. Starting from Android 5, the webview is shipped as a separate component with updates. Embedding Crosswalk is still convenient as you know exactly which version you can rely on. 

  2. I hope to have this fixed in later versions. 

  3. This may seem harmful and you are right. However, if you have an internal CA, it is currently not possible to provide its own trust store to a webview. Moreover, the system trust store is not used either. You also may want to use TLS for authentication only with client certificates, a feature supported by Dashkiosk

  4. Crosswalk being an opensource project, an alternative would have been to patch Crosswalk source code and recompile it. However, Crosswalk embeds Chromium and recompiling the whole stuff consumes a lot of resources. 

03 December, 2016 10:20PM by Vincent Bernat

December 02, 2016

hackergotchi for Shirish Agarwal

Shirish Agarwal

Air Congestion and Politics

Confession time first – I am not a frequent flyer at all. My first flight was in early late 2006. It was a 2 hour flight from Bombay (BOM) to Bengaluru (formerly Bangalore, BLG) . I still remember the trepidation, the nervousness and excitement the first time I took to air. I still remember the flight very vividly,

It was a typical humid day for Bombay/Mumbai and we (me and a friend) had gone to Sahar (the domestic airport) to take the flight in the evening. Before starting the sky had turned golden-orange and I was wondering how I would feel once I would be in air.We started at around 20:00 hours in the evening and as it was a clear night were able to see the Queen’s necklace (Marine Drive) in all her glory.

The photographs on the wikipedia page don’t really do justice to how beautiful the whole boulevard looks at night, especially how it looks from up there. While we were seeing, it seemed the pilot had actually banked at 45 degrees angle so we can have the best view of the necklace OR maybe the pilot wanted to take a photo OR ME being in overdrive (like Robin Williams, the Russian immigrant in Moscow on the Hudson experiences the first time he goes to the mall ;))

In either way, this would be an experience I would never forget till the rest of my life. I remember I didn’t move an inch (even to go the loo) as I didn’t want to let go of the whole experience. While I came back after 3-4 days, I still remember re-experiencing/re-imagining the flights for a whole month each time I went to sleep.

While I can’t say it has become routinised, but have been lucky to have the opportunity to fly domestic around the country primarily for work. After the initial romanticism wears off, you try and understand the various aspects of the flight which are happening around you.

These experiences are what lead to file/share today’s blog post. Yesterday, Ms. Mamata Banerjee, one of the leaders of the Opposition cried wolf because the Aircraft was circling the Airport. Because she is the Chief Minister she feels she should have got precedent or at least that seems to be the way the story unfolded on TV.

I have been about 15-20 times on flight in the last decade for work or leisure. Almost all the flights I have been, it has been routine that the flights fly around the Airport for 15-20 minutes before landing. This is ‘routine’. I have seen Airlines being stacked (remember the scene from Die Hard 2 where Holly Mclane, John Mclane’s wife looks at different aircraft at different altitudes from her window seat) this is what an Airport has to do when it doesn’t have enough runaways. In fact just read few days back MIAL is going for an emergency expansion as they weren’t expecting as many passengers as they did this year as well as last. In fact the same day there was a near-miss between two aircraft in Mumbai airport itself. Because of Ms. Mamata’s belligerence, this story didn’t even get a mention in the TV mainstream media.

The point I wanna underscore is that this is a fact of life and not just in India, world-over it seems hubs are being busier than ever, for instance Heathrow has been also a busy bee and they will to rework air operations as per a recent article .

In India, Kolkata is also one of the busier airports . If anything, I hope it teaches her the issues that plague most Indian airports and she works with the Government in Center so the Airport can expand more. They just got a new terminal three years back.

It is for these issues that the Indian Government has come with the ‘Regional Connectivity Scheme‘ .

Lastly, a bit of welcome news to people thinking to visit India, the Govt. of the day is facilitating easier visa norms to increase tourism and trade to India. Hope this is beneficial to all and any Debian Developers who wanna come visit India😉 I do hope that we also do get reciprocity from those countries as well.


Filed under: Miscellenous Tagged: # Domestic Flights, #Air Congestion, #Airport Expansion, #Kolkata, #near-miss, #Visa for tourists

02 December, 2016 03:20PM by shirishag75

hackergotchi for Raphaël Hertzog

Raphaël Hertzog

My Free Software Activities in November 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it’s one of the best ways to find volunteers to work with me on projects that matter to me.

Debian LTS

In the 11 hours of (paid) work I had to do, I managed to release DLA-716-1 aka tiff 4.0.2-6+deb7u8 fixing CVE-2016-9273, CVE-2016-9297 and CVE-2016-9532. It looks like this package is currently getting new CVE every month.

Then I spent quite some time to review all the entries in dla-needed.txt. I wanted to get rid of some misleading/no longer applicable comments and at the same time help Olaf who was doing LTS frontdesk work for the first time. I ended up tagging quite a few issues as no-dsa (meaning that we will do nothing for them as they are not serious enough) such as those affecting dwarfutils, dokuwiki, irssi. I dropped libass since the open CVE is disputed and was triaged as unimportant. While doing this, I fixed a bug in the bin/review-update-needed script that we use to identify entries that have not made any progress lately.

Then I claimed libgc and and released DLA-721-1 aka libgc 1:7.1-9.1+deb7u1 fixing CVE-2016-9427. The patch was large and had to be manually backported as it was not applying cleanly.

The last thing I did was to test a new imagemagick and review the update prepared by Roberto.

pkg-security work

The pkg-security team is continuing its good work: I sponsored patator to get rid of a useless dependency on pycryptopp which was going to be removed from testing due to #841581. After looking at that bug, it turns out the bug was fixed in libcrypto++ 5.6.4-3 and I thus closed it.

I sponsored many uploads: polenum, acccheck, sucrack (minor updates), bbqsql (new package imported from Kali). A bit later I fixed some issues in the bbsql package that had been rejected from NEW.

I managed a few RC bugs related to the openssl 1.1 transition: I adopted sslsniff in the team and fixed #828557 by build-depending on libssl1.0-dev after having opened the proper upstream ticket. I did the same for ncrack and #844303 (upstream ticket here). Someone else took care of samdump2 but I still adopted the package in the pkg-security team as it is a security relevant package. I also made an NMU for axel and #829452 (it’s not pkg-security related but we still use it in Kali).

Misc Debian work

Django. I participated in the discussion about a change letting Django count the number of developers that use it. Such a change has privacy implications and the discussion sparked quite some interest both in Debian mailing lists and up to LWN.

On a more technical level, I uploaded version 1.8.16-1~bpo8+1 to jessie-backports (security release) and I fixed RC bug #844139 by backporting two upstream commits. This led to the 1.10.3-2 upload. I ensured that this was fixed in the 1.10.x upstream branch too.

dpkg and merged /usr. While reading debian-devel, I discovered dpkg bug #843073 that was threatening the merged-/usr feature. Since the bug was in code that I wrote a few years ago, and since Guillem was not interested in fixing it, I spent an hour to craft a relatively clean patch that Guillem could apply. Unfortunately, Guillem did not yet manage to pull out a new dpkg release with the patches applied. Hopefully it won’t be too long until this happens.

Debian Live. I closed #844332 which was a request to remove live-build from Debian. While it was marked as orphaned, I was always keeping an eye on it and have been pushing small fixes to git. This time I decided to officially adopt the package within the debian-live team and work a bit more on it. I reviewed all pending patches in the BTS and pushed many changes to git. I still have some pending changes to finish to prettify the Grub menu but I plan to upload a new version really soon now.

Misc bugs filed. I filed two upstream tickets on uwsgi to help fix currently open RC bugs on the package. I filed #844583 on sbuild to support arbitrary version suffix for binary rebuild (binNMU). And I filed #845741 on xserver-xorg-video-qxl to get it fixed for the xorg 1.19 transition.

Zim. While trying to fix #834405 and update the required dependencies, I discovered that I had to update pygtkspellcheck first. Unfortunately, its package maintainer was MIA (missing in action) so I adopted it first as part of the python-modules team.

Distro Tracker. I fixed a small bug that resulted in an ugly traceback when we got queries with a non-ASCII HTTP_REFERER.

Thanks

See you next month for a new summary of my activities.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

02 December, 2016 11:45AM by Raphaël Hertzog

hackergotchi for Matthew Garrett

Matthew Garrett

Ubuntu still isn't free software

Mark Shuttleworth just blogged about their stance against unofficial Ubuntu images. The assertion is that a cloud hoster is providing unofficial and modified Ubuntu images, and that these images are meaningfully different from upstream Ubuntu in terms of their functionality and security. Users are attempting to make use of these images, are finding that they don't work properly and are assuming that Ubuntu is a shoddy product. This is an entirely legitimate concern, and if Canonical are acting to reduce user confusion then they should be commended for that.

The appropriate means to handle this kind of issue is trademark law. If someone claims that something is Ubuntu when it isn't, that's probably an infringement of the trademark and it's entirely reasonable for the trademark owner to take action to protect the value associated with their trademark. But Canonical's IP policy goes much further than that - it can be interpreted as meaning[1] that you can't distribute works based on Ubuntu without paying Canonical for the privilege, even if you call it something other than Ubuntu.

This remains incompatible with the principles of free software. The freedom to take someone else's work and redistribute it is a vital part of the four freedoms. It's legitimate for Canonical to insist that you not pass it off as their work when doing so, but their IP policy continues to insist that you remove all references to Canonical's trademarks even if their use would not infringe trademark law.

If you ask a copyright holder if you can give a copy of their work to someone else (assuming it doesn't infringe trademark law), and they say no or insist you need an additional contract, it's not free software. If they insist that you recompile source code before you can give copies to someone else, it's not free software. Asking that you remove trademarks that would otherwise infringe trademark law is fine, but if you can't use their trademarks in non-infringing ways, that's still not free software.

Canonical's IP policy continues to impose restrictions on all of these things, and therefore Ubuntu is not free software.

[1] And by "interpreted as meaning" I mean that's what it says and Canonical refuse to say otherwise

comment count unavailable comments

02 December, 2016 09:37AM

December 01, 2016

Thorsten Alteholz

My Debian Activities in November 2016

FTP assistant

This month I marked 377 packages for accept and rejected 36 packages. I also sent 13 emails to maintainers asking questions.

Debian LTS

This was my twenty-ninth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 11h. During that time I did uploads of

  • [DLA 696-1] bind9 security update for one CVE
  • [DLA 711-1] curl security update for nine CVEs

The upload of curl started as an embargoed one but the discussion about one fix took some time and the upload was a bit delayed.

I also prepared a test package for jasper which takes care of nine CVEs and is available here. If you are interested in jasper, please download it and check whether everything is working in your environment. As upstream only takes care of CVEs/bugs at the moment, maybe we should not upload the old version with patches but the new version with all fixes. Any comments?

Other stuff

As it is again this time of the year, I would also like to draw some attention to the Debian Med Advent Calendar. Like the past years, the Debian Med team starts a bug squashing event from the December 1st to 24th. Every bug that is closed will be registered in the calendar. So instead of taking something from the calendar, this special one will be filled and at Christmas hopefully every Debian Med related bug is closed. Don’t hestitate, start to squash :-).

In November I also uploaded new versions of libmatthew-java, node-array-find-index, node-ejs, node-querystringify, node-require-dir, node-setimmediate, libkeepalive,
Further I added node-json5, node-emojis-list, node-big.js, node-eslint-plugin-flowtype to the NEW queue, sponsored an upload of node-lodash, adopted gnupg-pkcs11-scd, reverted the -fPIC-patch in libctl and fixed RC bugs in alljoyn-core-1504, alljoyn-core-1509, alljoyn-core-1604.

01 December, 2016 09:33PM by alteholz

hackergotchi for Daniel Pocock

Daniel Pocock

Using a fully free OS for devices in the home

There are more and more devices around the home (and in many small offices) running a GNU/Linux-based firmware. Consider routers, entry-level NAS appliances, smart phones and home entertainment boxes.

More and more people are coming to realize that there is a lack of security updates for these devices and a big risk that the proprietary parts of the code are either very badly engineered (if you don't plan to release your code, why code it properly?) or deliberately includes spyware that calls home to the vendor, ISP or other third parties. IoT botnet incidents, which are becoming more widely publicized, emphasize some of these risks.

On top of this is the frustration of trying to become familiar with numerous different web interfaces (for your own devices and those of any friends and family members you give assistance to) and the fact that many of these devices have very limited feature sets.

Many people hail OpenWRT as an example of a free alternative (for routers), but I recently discovered that OpenWRT's web interface won't let me enable both DHCP and DHCPv6 concurrently. The underlying OS and utilities fully support dual stack, but the UI designers haven't encountered that configuration before. Conclusion: move to a device running a full OS, probably Debian-based, but I would consider BSD-based solutions too.

For many people, the benefit of this strategy is simple: use the same skills across all the different devices, at home and in a professional capacity. Get rapid access to security updates. Install extra packages or enable extra features if really necessary. For example, I already use Shorewall and strongSwan on various Debian boxes and I find it more convenient to configure firewall zones using Shorewall syntax rather than OpenWRT's UI.

Which boxes to start with?

There are various considerations when going down this path:

  • Start with existing hardware, or buy new devices that are easier to re-flash? Sometimes there are other reasons to buy new hardware, for example, when upgrading a broadband connection to Gigabit or when an older NAS gets a noisy fan or struggles with SSD performance and in these cases, the decision about what to buy can be limited to those devices that are optimal for replacing the OS.
  • How will the device be supported? Can other non-technical users do troubleshooting? If mixing and matching components, how will faults be identified? If buying a purpose-built NAS box and the CPU board fails, will the vendor provide next day replacement, or could it be gone for a month? Is it better to use generic components that you can replace yourself?
  • Is a completely silent/fanless solution necessary?
  • Is it possibly to completely avoid embedded microcode and firmware?
  • How many other free software developers are using the same box, or will you be first?

Discussing these options

I recently started threads on the debian-user mailing list discussing options for routers and home NAS boxes. A range of interesting suggestions have already appeared, it would be great to see any other ideas that people have about these choices.

01 December, 2016 01:11PM by Daniel.Pocock

November 30, 2016

Carl Chenet

My Free Software activities in November 2016

My Monthly report for Novembre 2016 gives an extended list of what were my Free Software related activities during this month.

Personal projects:

Journal du hacker:

The Journal du hacker is a frenck-speaking Hacker News-like website dedicated to the french-speaking Free and Open source Software community.

logo-journal-du-hacker

That’s all folks! See you next month!

30 November, 2016 11:00PM by Carl Chenet

hackergotchi for Joey Hess

Joey Hess

drought

Drought here since August. The small cistern ran dry a month ago, which has never happened before. The large cistern was down to some 900 gallons. I don't use anywhere near the national average of 400 gallons per day. More like 10 gallons. So could have managed for a few more months. Still, this was worrying, especially as the area moved from severe to extreme drought according to the US Drought Monitor.

Two days of solid rain fixed it, yay! The small cistern has already refilled, and the large will probably be full by tomorrow.

The winds preceeding that same rain storm fanned the flames that destroyed Gatlinburg. Earlier, fire got within 10 miles of here, although that may have been some kind of controlled burn.

Climate change is leading to longer duration weather events in this area. What tended to be a couple of dry weeks in the fall, has become multiple months of drought and weeks of fire. What might have been a few days of winter weather and a few inches of snow before the front moved through has turned into multiple weeks of arctic air, with multiple 1 ft snowfalls. What might have been a few scorching summer days has become a week of 100-110 degree temperatures. I've seen all that over the past several years.

After this, I'm adding "additional, larger cistern" to my todo list. Also "larger fire break around house".

30 November, 2016 10:09PM

hackergotchi for Chris Lamb

Chris Lamb

Free software activities in November 2016

Here is my monthly update covering what I have been doing in the free software world (previous month):

  • Started work on a Python API to the UK Postbox mail scanning and forwarding service. (repo)
  • Lots of improvements to buildinfo.debian.net, my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them, including making GPG signatures mandatory (#7), updating jenkins.debian.net to sign them and moving to SSL.
  • Improved the Django client to the KeyError error tracking software, enlarging the test coverage and additionally adding support for grouping errors using a context manager.
  • Made a number of improvements to travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change:
    • Install build-dependencies with debugging output. Thanks to @waja. (#31)
    • Install Lintian by default. Thanks to @freeekanayaka. (#33).
    • Call mktemp with --dry-run to avoid having to delete it later. (commit)
  • Submitted a pull request to Wheel (a utility to package Python libraries) to make the output of METADATA files reproducible. (#73)
  • Submitted some miscellaneous documentation updates to the Tails operating system. (patches)

Reproducible builds


Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.


This month:


My work in the Reproducible Builds project was also covered in our weekly reports. (#80, #81, #82 #83.


Toolchain issues

I submitted the following patches to fix reproducibility-related toolchain issues with Debian:


strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.


jenkins.debian.net

jenkins.debian.net runs our comprehensive testing framework.

  • buildinfo.debian.net has moved to SSL. (ac3b9e7)
  • Submit signing keys to keyservers after generation. (bdee6ff)
  • Various cosmetic changes, including
    • Prefer if X not in Y over if not X in Y. (bc23884)
    • No need for a dictionary; let's just use a set. (bf3fb6c)
    • Avoid DRY violation by using a for loop. (4125ec5)

I also submitted 9 patches to fix specific reproducibility issues in apktool, cairo-5c, lava-dispatcher, lava-server, node-rimraf, perlbrew, qsynth, tunnelx & zp.


Debian


Debian LTS

This month I have been paid to work 11 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 697-1 for bsdiff fixing an arbitrary write vulnerability.
  • Issued DLA 705-1 for python-imaging correcting a number of memory overflow issues.
  • Issued DLA 713-1 for sniffit where a buffer overflow allowed a specially-crafted configuration file to provide a root shell.
  • Issued DLA 723-1 for libsoap-lite-perl preventing a Billion Laughs XML expansion attack.
  • Issued DLA 724-1 for mcabber fixing a roster push attack.

Uploads

  • redis:
    • 3.2.5-2 — Tighten permissions of /var/{lib,log}/redis. (#842987)
    • 3.2.5-3 & 3.2.5-4 — Improve autopkgtest tests and install upstream's MANIFESTO and README.md documentation.
  • gunicorn (19.6.0-9) — Adding autopkgtest tests.
  • libfiu:
    • 0.94-1 — Add autopkgtest tests.
    • 0.95-1, 0.95-2 & 0.95-3 — New upstream release and improve autopkgtest coverage.
  • python-django (1.10.3-1) — New upstream release.
  • aptfs (0.8-3, 0.8-4 & 0.8-5) — Adding and subsequently improving the autopkgtext tests.


I performed the following QA uploads:



Finally, I also made the following non-maintainer uploads:

  • libident (0.22-3.1) — Move from obsolete Source-Version substvar to binary:Version. (#833195)
  • libpcl1 (1.6-1.1) — Move from obsolete Source-Version substvar to binary:Version. (#833196)
  • pygopherd (2.0.18.4+nmu1) — Move from obsolete Source-Version substvar to ${source:Version}. (#833202)


RC bugs



I also filed 59 FTBFS bugs against arc-gui-clients, asyncpg, blhc, civicrm, d-feet, dpdk, fbpanel, freeciv, freeplane, gant, golang-github-googleapis-gax-go, golang-github-googleapis-proto-client-go, haskell-cabal-install, haskell-fail, haskell-monadcatchio-transformers, hg-git, htsjdk, hyperscan, jasperreports, json-simple, keystone, koji, libapache-mod-musicindex, libcoap, libdr-tarantool-perl, libmath-bigint-gmp-perl, libpng1.6, link-grammar, lua-sql, mediatomb, mitmproxy, ncrack, net-tools, node-dateformat, node-fuzzaldrin-plus, node-nopt, open-infrastructure-system-images, open-infrastructure-system-images, photofloat, ppp, ptlib, python-mpop, python-mysqldb, python-passlib, python-protobix, python-ttystatus, redland, ros-message-generation, ruby-ethon, ruby-nokogiri, salt-formula-ceilometer, spykeviewer, sssd, suil, torus-trooper, trash-cli, twisted-web2, uftp & wide-dhcpv6.


FTP Team


As a Debian FTP assistant I ACCEPTed 70 packages: bbqsql, coz-profiler, cross-toolchain-base, cross-toolchain-base-ports, dgit-test-dummy, django-anymail, django-hstore, django-html-sanitizer, django-impersonate, django-wkhtmltopdf, gcc-6-cross, gcc-defaults, gnome-shell-extension-dashtodock, golang-defaults, golang-github-btcsuite-fastsha256, golang-github-dnephin-cobra, golang-github-docker-go-events, golang-github-gogits-cron, golang-github-opencontainers-image-spec, haskell-debian, kpmcore, libdancer-logger-syslog-perl, libmoox-buildargs-perl, libmoox-role-cloneset-perl, libreoffice, linux-firmware-raspi3, linux-latest, node-babel-runtime, node-big.js, node-buffer-shims, node-charm, node-cliui, node-core-js, node-cpr, node-difflet, node-doctrine, node-duplexer2, node-emojis-list, node-eslint-plugin-flowtype, node-everything.js, node-execa, node-grunt-contrib-coffee, node-grunt-contrib-concat, node-jquery-textcomplete, node-js-tokens, node-json5, node-jsonfile, node-marked-man, node-os-locale, node-sparkles, node-tap-parser, node-time-stamp, node-wrap-ansi, ooniprobe, policycoreutils, pybind11, pygresql, pysynphot, python-axolotl, python-drizzle, python-geoip2, python-mockupdb, python-pyforge, python-sentinels, python-waiting, pythonmagick, r-cran-isocodes, ruby-unicode-display-width, suricata & voctomix-outcasts.

I additionally filed 4 RC bugs against packages that had incomplete debian/copyright files against node-cliui, node-core-js, node-cpr & node-grunt-contrib-concat.

30 November, 2016 09:18PM

Jonas Meurer

debian lts report 2016.11

Debian LTS report for November 2016

Noevember 2016 was my third month as a Debian LTS team member. I was allocated 11 hours and had 1,75 hours left from October. This makes a total of 12,75 hours. In November I spent all 12,75 hours (and even a bit more) preparing security updates for spip, memcached and monit.

In particular, the updates of spip and monit took a lot of time (each one more than six hours). The patches for both packages were horrible to backport as the affected codebase changed a lot between the Wheezy versions and current upstream versions. Still it was great fun and I learned a lot during the backporting work. Due to the intrusive nature of the patches, I also did much more extensive testing before uploading the packages, which took quite a bit of time as well.

Monit 5.4-2+deb7u1 is not uploaded to wheezy-security yet as I decided to ask for further review and testing on the debian-lts mailinglist first.

Below follows the list of items I worked on in November in the well known format:

  • DLA 695-1: several XSS, CSRF and code execution flaws fixed in spip 2.1.17-1+deb7u6
  • DLA 701-1: integer overflows, buffer over-read fixed in memcached 1.4.13-0.2+deb7u2
  • CVE-2016-7067: backported CSRF protection to monit 5.4-2+deb7u1

30 November, 2016 07:34PM

Arturo Borrero González

Creating a team for netfilter packages in debian

Debian - Netfilter

There are about 15 Netfilter packages in Debian, and they are maintained by separate people.

Yersterday, I contacted the maintainers of the main packages to propose the creation of a pkg-netfilter team to maintain all the packages together.

The benefits of maintaining packages in a team is already known to all, and I would expect to rise the overall quality of the packages due to this movement.

By now, the involved packages and maintainers are:

We should probably ping Jochen Friedrich as well who maintains arptables and ebtables. Also, there are some other non-official Netfilter packages, like iptables-persistent. I’m undecided to what to do with them, as my primary impulse is to only put in the team upstream packages.

Given the release of Stretch is just some months ahead, the creation of this packaging team will happen after the release, so we don’t have any hurry moving things now.

30 November, 2016 05:00AM

November 29, 2016

hackergotchi for Shirish Agarwal

Shirish Agarwal

The Iziko South African Museum

This would be a bit long on my stay in Cape Town, South Africa after Debconf16.

Before I start, let me share the gallery works, you can see some photos that I have been able to upload to my gallery . It seems we are using gallery 2 while upstream had made gallery 3 and then it sort of died. I actually asked in softwarerecs stackexchange site if somebody knows of a drop-in replacement for gallery and was told/shared about Pwigo . I am sure the admin knows about it. There would be costs to probably migrate from gallery to Pwigo with the only benefit that it would be something which would perhaps be more maintainable.

The issues I face with the current gallery system are few things –

a. There is no way to know how much your progress your upload has taken.
b. After it has submit, it gives a fake error message saying some error has occurred. This has happened on every occasion/attempt. Now I don’t know whether it is because I have slow upload speeds or something else altogether. I had shared the error page last time in the blog post hence not sharing again.

Although, all the pictures which would be shared in this blog post would be from the same gallery🙂

Another thing I would like to share is a small beginner article I wrote about why I like Debian.

Another interesting/tit-bit of news I came to know few days back that both Singapore and Qatar have given 96 hours visa free stopovers for Indians for select destinations.

Now to start with the story/experience due to some unknown miracle/angel looking upon me I got the chance to go to Debconf16, South Africa. I’m sure there was lot of backend discussions but in the end I was given the opportunity to be part of Debcamp and Debconf. While I hope to recount my Debcamp and Debconf experience in another or two blog posts, this would be exclusively the Post-Debconf Experiences I had.

As such opportunities to visit another country are rare, I wanted to make the most of it. Before starting from Pune, I had talked with Amey about Visas, about Debconf as he had just been to Debconf15 the year before and various things related to travel. He was instrumental in me having a bit more knowledge about how to approach things. I was also lucky to have both Graham and Bernelle who also suggested, advised and made it possible to have a pleasant stay both during Debcamp and Debconf. The only quibble is I didn’t know heaters were being made available to us without any cost.

Moving on, a day or two before Debconf was about to conclude, I asked Bernelle’s help even though she was battling a burn-out I believe as I was totally clueless about Cape Town. She accepted my request and asked me to look at hostels near Longmarket Street. I had two conditions –

a. It should not be very far from the airport
b. It should be near to all or most cultural experiences the city has to offer.

We looked at hostelworld and from the options listed, it looked like Homebasecapetown looked to be a perfect fit. It was one of the cheaper options and they also had breakfast included in the pricing. I booked through hostelworld for a mixed dorm for 2 days as I was unsure how it would be (the first night effect I have shared about previously) .

When I reached there, I found it to be as good as the pictures shared were, the dorm was clean (most important), people were friendly (also important) as well as toilets and shower were also clean while the water was hot, so all in all it was a win-win situation for me.

Posters I saw at homebasecapetown

While I’m not much of an adrenaline-junkie it was nice to know the activities that could be done/taken.

Brochures and Condoms just left of main hall.

This was again interesting. While apologies for the poor shaky quality of the picture, I believe it is easy to figure out. There were Brochures of the city attractions as well as condoms that people could discreetly use if need be. I had seen such condoms in few toilets during and around Debconf and it felt good that the public were aware and prioritizing safety for their guests and students instead of having fake holier than thou attitudes that many places have.

For instance, you wouldn’t find something like this in toilets of most colleges in India or anywhere else for that matter. There are few vending machines in what are termed as ‘red light areas’ or where prostitution is known/infamous to happen and even then most times it is empty. I have 2-3 social workers as friends and they are a source of news on such things.

While I went to few places and each had an attraction to it, the one which had my literally eyes out of socket was the ‘Iziko South African Museum‘ . I have been lucky to been quite a few museums in India, the best rated science museum in India in my limited experience has been the ‘Visvesvaraya Industrial & Technological Museum, Bengaluru – India‘. A beer from me if a European can get it right.

Don’t worry if you mispronounce it, I mispronounce it couple of times till I get it right🙂 .

Looking up the word ‘Iziko’ the meaning of the word seems to be ‘the hearth’ and if you look at the range of collections in the museum, you would think it fits.

I was lucky to find couple of friends, one of whom was living at homebase and we decided to go to the museum together.

Making friends on the road

So Eduardo, my friend on the left and his friend, we went to the museum. While viewing the museum, there were no adjectives to describe it other than ‘Wow’ and ‘Endless’ .

See –

fossils of fish-whale-shark ?

OR

Giant fish-whale-dolphin-shark some million years ago.

and

Reminder of JAWS ;)

While I have more than a few pictures, the point is easily made. It seems almost inconceivable that creatures of such masses actually were on earth. While I played with the model of the jaws of a whale/shark in reality if something like that happened, I would have been fighting for my life.

The only thing I missed or could have been better if they had some interactive installations to showcase the now universally accepted Charles Darwin’s ‘On the Origin of Species‘ I had never seen anything like this. Sadly, there was nobody around to help us figure out things as I had read that most species of fish don’t leave a skeleton behind so how were these models made? It just boggles the mind.

Apart from the Science Museum I was also introduced to the bloody history that South Africa had. I saw –

The 1913 native land act which was not honored .

I had been under the impression that India had got a raw deal when it was under British rule but looking at South African history I don’t know. While we got our freedom in 1947 they got rid of apartheid about 20 years+ . I talked to lot of young African males and there was lot of naked hostility for the Europeans even today. It was a bit depressing but could relate to their point of view as similar sentiments were echoed by our forefathers. I read in the newspapers and it seemed to be a pretty mixed picture.

I can’t comment as only South Africans can figure out the way forward. For me, it was enough to know and see that we both had similar political histories as nations. It seemed the racial divide and anger was much more highly pronounced towards Europeans and divisive then the caste divisions here between Indians. I also shared with them my limited knowledge and understanding of the Indian history (as history is re-written all the time) and it was clear to them that we had common/similar pasts.

As a result, what was surprising (actually not) is that many South Africans have no knowledge of Indian history. as well otherwise the political differences that South Africa and India has in the current scenario wouldn’t have been.

In the end, the trip proved to be fun, stimulating, educative, thought-provoking as questions about self-identity , national identity, our place in the Universe kinda questions which should be asked all the time.

Thank you Bremmer and the team for letting me experience Cape Town, South Africa, I would have been poorer if I hadn’t had the experience.


Filed under: Miscellenous Tagged: #Debconf16, #Dinosaur Fishes, #gallery, #Identity, #Iziko South African Museum, #Nation-state Identity, #pwigo

29 November, 2016 08:49PM by shirishag75

Reproducible builds folks

Reproducible Builds: week 83 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 20 and Saturday November 26 2016:

Reproducible work in other projects

Bugs filed

Chris Lamb:

Daniel Shahaf:

Reiner Herrmann:

Reviews of unreproducible packages

63 package reviews have been added, 73 have been updated and 41 have been removed in this week, adding to our knowledge about identified issues.

4 issue types have been added:

Weekly QA work

During our reproducibility testing, some FTBFS bugs have been detected and reported by:

  • Chris Lamb (9)
  • Helmut Grohne (1)
  • Peter De Wachter (1)

strip-nondeterminism development

  • #845203 was fixed in git by Reiner Herrmann - the next release will be able to normalize NTFS timestamps in zip files.

debrepatch development

Continuous integration:

  • Holger updated our jenkins jobs for disorderfs and strip-nondeterminism to build these from their respective git master branches, and removed the jobs that build them from other branches since we have none at the moment.

tests.reproducible-builds.org

Debian:

Since the stretch freeze is getting closer, Holger made the following changes:

  • Schedule testing builds to be as equally-frequent as unstable, on all archs, so that testing's build results are more up-to-date.

  • Adjust experimental builds scheduling frequency so that experimental results are not more recent than the ones in unstable.

  • Disable our APT repository for the testing suite (stretch), but leave it active for the unstable and experimental suites.

    This is the repository where we uploaded patched toolchain packages from time to time, that are necessary to reproduce other packages with. Since recently, all our essential patches have been accepted into Debian stretch and this repository is currently empty. Debian stretch will soon become the next Debian stable, and we want to get an accurate impression of how many of its packages will be reproducible.

    Therefore, disabling this repository for stretch whilst leaving it activated for the Debian unstable and experimental suites, allows us to continue to experiment with new patches to toolchain packages, without affecting our knowledge of the next Debian stable.

Misc.

This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

29 November, 2016 06:12PM

Mike Hommey

Announcing git-cinnabar 0.4.0 release candidate

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.4.0b3?

  • Updated git to 2.10.2 for cinnabar-helper.
  • Added a new git cinnabar download command to download a helper on platforms where one is available.
  • Fixed some corner cases with pack windows in the helper. This prevented cloning mozilla-central with the helper.
  • Fixed bundle2 support that broke cloning from a mercurial 4.0 server in some cases.
  • Fixed some corner cases involving empty files. This prevented cloning Mozilla’s stylo incubator repository.
  • Fixed some correctness issues in file parenting when pushing changesets pulled from one mercurial repository to another.
  • Various improvements to the rules to build the helper.
  • Experimental (and slow) support for pushing merges, with caveats. See issue #20 for details about the current status.

And since I realize I didn’t announce beta 3:

What’s new since 0.4.0b2?

  • Properly handle bundle2 errors, avoiding git to believe a push happened when it didn’t. (0.3.x is unaffected)

29 November, 2016 12:18AM by glandium

November 28, 2016

hackergotchi for Michal Čihař

Michal Čihař

phpMyAdmin security issues

You might wonder why there is so high number of phpMyAdmin security announcements this year. This situations has two main reasons and I will comment a bit on those.

First of all we've got quite a lot of attention of people doing security reviews this year. It has all started with Mozilla SOS Fund funded audit. It has discovered few minor issues which were fixed in the 4.6.2 release. However this was really just the beginning of the story and the announcement has attracted quite some attention to us. In upcoming weeks the security@phpmyadmin.net mailbox was full of reports and we really struggled to handle such amount. Handling that amount actually lead to creating more formalized approach to handling them as we clearly were no longer able to deal with them based on email only. Anyway most work here was done by Emanuel Bronshtein, who is really looking at every piece of our code and giving useful tips to harden our code base and infrastructure.

Second thing which got changed is that we release security announcements for security hardening even when there might not be any practical attack possible. Typical example here might be PMASA-2016-61, where using hash_equals is definitely safer, but even if the timing attack would be doable here, the practical result of figuring out admin configured allow/deny rules is usually not critical. Many of the issues also cover quite rare setups (or server misconfigurations, which we've silently fixed in past) like PMASA-2016-54 being possibly caused by server executing shell scripts shipped together with phpMyAdmin.

Overall phpMyAdmin indeed got safer this year. I don't think that there was any bug that would be really critical, on the other side we've made quite a lot of hardenings and we use current best practices when dealing with sensitive data. On the other side, I'm pretty sure our code was not in worse shape than any similarly sized projects with 18 years of history, we just become more visible thanks to security audit and people looked deeper into our code base.

Besides security announcements this all lead to generic hardening of our code and infrastructure, what might be not that visible, but are important as well:

  • All our websites are server by https only
  • All our releases are PGP signed
  • We actively encourage users to verify the downloaded files
  • All new Git tags are PGP signed as well

Filed under: Debian English phpMyAdmin SUSE | 0 comments

28 November, 2016 05:00PM

hackergotchi for Clint Adams

Clint Adams

Not the Grace Hopper Conference

Do you love porting? For ideas on how to make GHC suck less on your favorite architecture, see this not-at-all ugly table.

28 November, 2016 01:56PM

Stefano Zacchiroli

last week to take part in the Debian Contributors Survey

Debian Contributors Survey 2016

About 3 weeks ago, together with Molly and Mathieu, we launched the first edition of the Debian Contributors Survey. I won't harp on it any further, because you can find all relevant information about it on the Debian blog or as part of the original announcement.

But it's worth noting that you've now only one week left to participate if you want to: the deadline for participation is 4 December 2016, at 23:59 UTC.

If you're a Debian contributor and would like to participate, just go to the survey participation page and fill in!

28 November, 2016 10:27AM

hackergotchi for Pau Garcia i Quiles

Pau Garcia i Quiles

Desktops DevRoom @ FOSDEM 2017: you are still on time to submit a talk

FOSDEM 2016 is going to be great (again!) and you still have the chance to be one of the stars.

Have you submitted your talk to the Desktops DevRoom yet?

No?

Remember: we will only accept proposals until December 5th. After that, the Organization Team will get busy and vote and choose the talks.

Here is the full Call for Participation, in case you need to check the details on how to submit:

FOSDEM Desktops DevRoom 2017 Call for Participation

Topics include anything related to the Desktop: desktop environments, software development for desktop/cross-platform, applications, UI, etc

28 November, 2016 12:24AM by pgquiles

November 27, 2016

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

anytime 0.1.1: More robust

CRAN just accepted the newest release 0.1.1 of anytime, following the previous five releases since September.

anytime is a very focussed package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects -- and to do so without requiring a format string.

See the anytime page, or the GitHub README.md for a few examples, or just consider the following illustration:

R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R> 

Release 0.1.1 robustifies two aspects. The 'digits only' input above extends what Boost Date_Time can parse and relies on simple-enough pre-processing. This operation is now more robust. We also ensure that input already of class Date is simply passed through by anydate() or utcdate(). Last but not least we added code coverage support, which oh-so-predictably lead us to game this metric to reach the elusive 100% coverage.

The NEWS file summarises the release:

Changes in anytime version 0.1.1 (2016-11-27)

  • Both anydate() and utcdate() no longer attempt to convert an input value that is already of type Date.

  • The string splitter (needed for the 'all-digits' formats extending Boost Date_time) is now more defensive about the input argument and more robust. Thanks to Bob Jansen for the heads-up (PR #30 closing issue #29).

  • Code coverage reporting has been added (PR #31).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 November, 2016 09:09PM

hackergotchi for Eriberto Mota

Eriberto Mota

Debian with three monitors under low cost graphics interface

Since 2008 I use two monitors in my desktop. Yesterday I bought a new graphics interface and a third monitor. Some time I was looking for a low cost graphics interface. Ok, I am using GeForce GT 740 which has three output ports: VGA, DVI and HDMI. In Brazil this interface card can be found around R$ 400 (US$ 117, but my card was US$ 87 in Brazilian Black Friday). In Amazon.com, it is between US$ 51 and US$ 109. The chosen manufacturer was Zotac, but all GT 740 and 750 will work fine (I tested the GT 750 too).

The GeForce GT 740 was imediatelly recognised by Debian Jessie with kernel Linux 4.7.0 from Backports (it is my default, so I didn't test with original 3.16 kernel). The driver used was the default X.Org Nouveau. I use KDE and the management was easy.

I hope this post can help people interested in use 3 monitors. Enjoy!

 

03-monitors

27 November, 2016 06:27PM by Eriberto

November 25, 2016

Julian Andres Klode

Starting the faster, more secure APT 1.4 series

We just released the first beta of APT 1.4 to Debian unstable (beta here means that we don’t know any other big stuff to add to it, but are still open to further extensions). This is the release series that will be released with Debian stretch, Ubuntu zesty, and possibly Ubuntu zesty+1 (if the Debian freeze takes a very long time, even zesty+2 is possible). It should reach the master archive in a few hours, and your mirrors shortly after that.

Security changes

APT 1.4 by default disables support for repositories signed with SHA1 keys. I announced back in January that it was my intention to do this during the summer for development releases, but I only remembered the Jan 1st deadline for stable releases supporting that (APT 1.2 and 1.3), so better late than never.

Around January 1st, the same or a similar change will occur in the APT 1.2 and 1.3 series in Ubuntu 16.04 and 16.10 (subject to approval by Ubuntu’s release team). This should mean that repository provides had about one year to fix their repositories, and more than 8 months since the release of 16.04. I believe that 8 months is a reasonable time frame to upgrade a repository signing key, and hope that providers who have not updated their repositories yet will do so as soon as possible.

Performance work

APT 1.4 provides a 10-20% performance increase in cache generation (and according to callgrind, we went from approx 6.8 billion to 5.3 billion instructions for my laptop’s configuration, a reduction of more than 21%). The major improvements are:

We switched the parsing of Deb822 files (such as Packages files) to my perfect hash function TrieHash. TrieHash – which generates C code from a set of words – is about equal or twice as fast as the previously used hash function (and two to three times faster than gperf), and we save an additional 50% of that time as we only have to hash once during parsing now, instead of during look up as well. APT 1.4 marks the first time TrieHash is used in any software. I hope that it will spread to dpkg and other software at a later point in time.vendors.

Another important change was to drop normalization of Description-MD5 values, the fields mapping a description in a Packages files to a translated description. We used to parse the hex digits into a native binary stream, and then compared it back to hex digits for comparisons, which cost us about 5% of the run time performance.

We also optimized one of our hash functions – the VersionHash that hashes the important fields of a package to recognize packages with the same version, but different content – to not normalize data to a temporary buffer anymore. This buffer has been the subject of some bugs (overflow, incompleteness) in the recent past, and also caused some slowdown due to the additional writes to the stack. Instead, we now pass the bytes we are interested in directly to our CRC code, one byte at a time.

There were also some other micro-optimisations: For example, the hash tables in the cache used to be ordered by standard compare (alphabetical followed by shortest). It is now ordered by size first, meaning we can avoid data comparisons for strings of different lengths. We also got rid of a std::string that cannot use short string optimisation in a hot path of the code. Finally, we also converted our case-insensitive djb hashes to not use a normal tolower_ascii(), but introduced tolower_ascii_unsafe() which just sets the “lowercase bit” (| 0x20) in the character.

Others

  • Sandboxing now removes some environment variables like TMP from the environment.
  • Several improvements to installation ordering.
  • Support for armored GPG keys in trusted.gpg.d.
  • Various other fixes

For a more complete overview of all changes, consult the changelog.


Filed under: Debian, Ubuntu

25 November, 2016 11:43PM by Julian Andres Klode

Petter Reinholdtsen

Quicker Debian installations using eatmydata

Two years ago, I did some experiments with eatmydata and the Debian installation system, observing how using eatmydata could speed up the installation quite a bit. My testing measured speedup around 20-40 percent for Debian Edu, where we install around 1000 packages from within the installer. The eatmydata package provide a way to disable/delay file system flushing. This is a bit risky in the general case, as files that should be stored on disk will stay only in memory a bit longer than expected, causing problems if a machine crashes at an inconvenient time. But for an installation, if the machine crashes during installation the process is normally restarted, and avoiding disk operations as much as possible to speed up the process make perfect sense.

I added code in the Debian Edu specific installation code to enable eatmydata, but did not have time to push it any further. But a few months ago I picked it up again and worked with the libeatmydata package maintainer Mattia Rizzolo to make it easier for everyone to get this installation speedup in Debian. Thanks to our cooperation There is now an eatmydata-udeb package in Debian testing and unstable, and simply enabling/installing it in debian-installer (d-i) is enough to get the quicker installations. It can be enabled using preseeding. The following untested kernel argument should do the trick:

preseed/early_command="anna-install eatmydata-udeb"

This should ask d-i to install the package inside the d-i environment early in the installation sequence. Having it installed in d-i in turn will make sure the relevant scripts are called just after debootstrap filled /target/ with the freshly installed Debian system to configure apt to run dpkg with eatmydata. This is enough to speed up the installation process. There is a proposal to extend the idea a bit further by using /etc/ld.so.preload instead of apt.conf, but I have not tested its impact.

25 November, 2016 01:50PM

hackergotchi for Iain R. Learmonth

Iain R. Learmonth

vmdebootstrap Sprint Report

This is now a little overdue, but here it is. On the 10th and 11th of November, the second vmdebootstrap sprint took place. Lars Wirzenius (liw), Ana Custura (ana_c) and myself were present. liw focussed on the core of vmdebootstrap, where he sketched out what the future of vmdebootstrap may look like. He documented this in a mailing list post and also presented (video).

Ana and myself worked on live-wrapper, which uses vmdebootstrap internally for the squashfs generation. I worked on improving logging, using a better method for getting paths within the image, enabling generation of Packages and Release files for the image archive and also made the images installable (live-wrapper 0.5 onwards will include an installer by default).

Ana worked on the inclusion of HDT and memtest86+ in the live images and enabled both ISOLINUX (for BIOS boot) and GRUB (for EFI boot) to boot the text-mode and graphical installers.

live-wrapper 0.5 was released on the 16th November with these fixes included. You can find live-wrapper documentation at https://live-wrapper.readthedocs.io/en/latest/. (The documentation still needs some work, some options may be incorrectly described).

Thanks to the sponsors that made this work possible. You’re awesome. (:

25 November, 2016 12:06PM

Michael Stapelberg

Debian package build tools

Personally, I find the packaging tools which are available in Debian far too complex. To better understand the options we have, I created a diagram of tools which are frequently used, only covering the build step (i.e. no post-build quality assurance checks or packaging-time helpers):

debian package build tools

When I was first introduced to Debian packaging, people recommended I use pbuilder. Given how complex the toolchain is in the pbuilder case, I don’t understand why that is (was?) a common recommendation.

Back in August 2015, so well over a year ago, I switched to sbuild, motivated by how much simpler it was to implement ratt (rebuilds reverse build dependencies) using sbuild, and I have not looked back.

Are there people who do not use sbuild for reasons other than familiarity? If so, please let me know, I’d like to understand.

I also made a version of the diagram above, colored by the programming languages in which the tools are implemented. The chosen colors are heavily biased :-).

debian package build tools, by language

To me, the diagram above means: if you want to make substantial changes to the Debian build tool infrastructure, you need to become an expert in all of Python, Perl, Bash, C and Make. I know that this is not true for every change, but it still irks me that there might be changes for which it is required.

I propose to eliminate complexity in Debian by deprecating the pbuilder toolchain in favor of sbuild.

25 November, 2016 11:30AM

November 24, 2016

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppExamples 0.1.8

A new version of the RcppExamples package is now on CRAN.

The RcppExamples package provides a handful of short examples detailing by concrete working examples how to set up basic R data structures in C++. This version takes advantage of the updated date and datetime classes in Rcpp 0.12.8 (which are optional for now and being phased in while we deprecate the old ones).

A NEWS extract follows:

Changes in RcppExamples version 0.1.8 (2016-11-24)

  • Updated DateExample to show vector addition available under Rcpp 0.12.8 when the (currently still phased in and optional) new Date(time) classes are used via the define in src/Makevars,.win; with fallback code for older versions

  • Other minor edits to DESCRIPTION and README.md

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 November, 2016 11:32PM

Vincent Fourmond

Finding zeros of data using QSoas

QSoas does not provide by default commands to detect zeros of data, and the reason for that is that it is simple, using the integrate command to convert this problem into a peak-finding problem, which can be solved using the find-peaks command. Here is that strategy applied to determining the zeros of the 0-th order bessel function:

QSoas> generate-buffer -10 10 bessel_j0(x) /samples=100001
QSoas> integrate
Current buffer now is: 'generated_int.dat'
QSoas> find-peaks
Found 6 peaks
buffer what x y index width left_width right_width
generated_int.dat min -8.6538 -0.201157042341714 6731 1.7798 0.905999999999999 0.873800000000001
generated_int.dat max -5.52 0.398165469321319 22400 2.2854 1.1862 1.0992
generated_int.dat min -2.4048 -0.403288737672291 37976 1.8232 0.973 0.850199999999999
generated_int.dat max 2.4048 2.53731134529594 62024 nan 2.2026 nan
generated_int.dat min 5.52 1.73585713830231 77600 nan 5.7198 nan
generated_int.dat max 8.6538 2.33517964996535 93269 nan 8.5532 nan

Compare that with the values given on Mathematica's website. This strategy is reasonably resistant to noise, since integration decreases high-frequency noise, but you may have to play with the /window option to find-peaks to avoid detecting the same zero (peak) several times.

Hopefully, I'll come back with more regular postings of tips and tricks !

24 November, 2016 09:23PM by Vincent Fourmond (noreply@blogger.com)

Sven Hoexter

first ditch effort - LyX 2.2.2 in unstable build with Qt5

No, not about the latest NOFX record, though it's a great one. Buy it. ;)

Took me a hell of a long time to get my head out of my arse and dive again into some Debian related work. Thanks to Nik for pushing me from time to time.

So I've taken the time to upload LyX 2.2.2 to unstable and it's now build with Qt5. Afterall the package is still missing a lot of love, but I hope we've once again something for the upcoming stable release, that is close to the latest upstream stable release. If you use LyX please give it a try.

For myself it's now the 6th year that I stopped using LyX after maintaining it for five years. And still I'm sponsoring the uploads and try to keep it at least functional. Strange how we sometimes take care of stuff even if we no longer have an active use for them.

24 November, 2016 08:07PM

hackergotchi for Ritesh Raj Sarraf

Ritesh Raj Sarraf

LIO -fb in Debian

LIO -fb is the new SCSI Target for Debian. Previously, we maintained the LIO tools from the pre-fork upstream branch. But, with good reasons, we've now moved to the newer -fb (Free Branch).

As the maintainer for those pacakges, I have a local LIO setup. Overy the years, I've been tuning and using this setup with a bunch of SCSI clients. Now with the new -fb packages it was worrisome for me, on how to migrate (Note: migration is not supported by the Debian packages) my old setup to the new one.

 

Thanks to Andy Grover for mentioning it, migrating your configuration is doable. With some minor intervention, I was able to switch my config from old LIO setup to the new LIO -fb pacakges. As you can see from the output below, both the outputs look the same, which is a good thing.

LIO reads its configuration from /etc/target/ and passes it into the kernel. The kernel loads the config. The real time config is present in configfs, within the kernel. Users willing for such migration need to ensure that the loaded config data remains in configfs. And then, using the new -fb tools (targetctl), the configuration data needs to be read and written to a new format in /etc/.

 

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- fileio ................................................................................................... [0 Storage Object]
  | o- iblock .................................................................................................. [4 Storage Objects]
  | | o- CENTOS ................................................................................................. [/dev/vdd, in use]
  | | o- SAN1 ................................................................................................... [/dev/vdb, in use]
  | | o- SAN2 ................................................................................................... [/dev/vdc, in use]
  | | o- SANROOT .............................................. [/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0, in use]
  | o- pscsi .................................................................................................... [0 Storage Object]
  | o- rd_mcp ................................................................................................... [0 Storage Object]
  o- ib_srpt ........................................................................................................... [0 Targets]
  o- iscsi ............................................................................................................. [3 Targets]
  | o- iqn.1994-05.com.redhat:23d8eb7fa1fc ................................................................................. [1 TPG]
  | | o- tpg1 ............................................................................................................ [enabled]
  | |   o- acls ............................................................................................................ [1 ACL]
  | |   | o- iqn.1994-05.com.redhat:23d8eb7fa1fc .................................................................... [1 Mapped LUN]
  | |   |   o- mapped_lun0 ............................................................................................. [lun0 (rw)]
  | |   o- luns ............................................................................................................ [1 LUN]
  | |   | o- lun0 ....................................................................................... [iblock/CENTOS (/dev/vdd)]
  | |   o- portals ..................................................................................................... [4 Portals]
  | |     o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
  | o- iqn.2003-01.org.linux-iscsi.debian.sanboot .......................................................................... [1 TPG]
  | | o- tpg1 ............................................................................................................ [enabled]
  | |   o- acls ............................................................................................................ [1 ACL]
  | |   | o- iqn.2005-03.org.open-iscsi:fd2bc2f4652a-sanboot ........................................................ [1 Mapped LUN]
  | |   |   o- mapped_lun0 ............................................................................................. [lun0 (rw)]
  | |   o- luns ............................................................................................................ [1 LUN]
  | |   | o- lun0 .................................... [iblock/SANROOT (/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0)]
  | |   o- portals ..................................................................................................... [4 Portals]
  | |     o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
  | |     o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
  | o- iqn.2003-01.org.linux-iscsi.debian.x8664 ............................................................................ [1 TPG]
  |   o- tpg1 ............................................................................................................ [enabled]
  |     o- acls ............................................................................................................ [1 ACL]
  |     | o- iqn.1993-08.org.debian:01:2972f6b5fc7 ................................................................. [2 Mapped LUNs]
  |     |   o- mapped_lun0 ............................................................................................. [lun0 (rw)]
  |     |   o- mapped_lun1 ............................................................................................. [lun1 (rw)]
  |     o- luns ........................................................................................................... [2 LUNs]
  |     | o- lun0 ......................................................................................... [iblock/SAN1 (/dev/vdb)]
  |     | o- lun1 ......................................................................................... [iblock/SAN2 (/dev/vdc)]
  |     o- portals ..................................................................................................... [4 Portals]
  |       o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
  |       o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
  |       o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
  |       o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
  o- loopback .......................................................................................................... [0 Targets]
  o- qla2xxx ........................................................................................................... [0 Targets]
  o- tcm_fc ............................................................................................................ [0 Targets]
  o- vhost ............................................................................................................. [0 Targets]
/> 







/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 4]
  | | o- CENTOS ........................................................................... [/dev/vdd (2.0GiB) write-thru activated]
  | | o- SAN1 ............................................................................. [/dev/vdb (1.0GiB) write-thru activated]
  | | o- SAN2 ............................................................................. [/dev/vdc (1.0GiB) write-thru activated]
  | | o- SANROOT ........................ [/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 (8.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 3]
  | o- iqn.1994-05.com.redhat:23d8eb7fa1fc ............................................................................... [TPGs: 1]
  | | o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  | |   o- acls .......................................................................................................... [ACLs: 1]
  | |   | o- iqn.1994-05.com.redhat:23d8eb7fa1fc .................................................................. [Mapped LUNs: 1]
  | |   |   o- mapped_lun0 ................................................................................ [lun0 block/CENTOS (rw)]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ........................................................................................ [block/CENTOS (/dev/vdd)]
  | |   o- portals .................................................................................................... [Portals: 4]
  | |     o- 172.16.20.40:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.41:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.42:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.43:3260 ................................................................................................ [OK]
  | o- iqn.2003-01.org.linux-iscsi.debian.sanboot ........................................................................ [TPGs: 1]
  | | o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  | |   o- acls .......................................................................................................... [ACLs: 1]
  | |   | o- iqn.2005-03.org.open-iscsi:fd2bc2f4652a-sanboot ...................................................... [Mapped LUNs: 1]
  | |   |   o- mapped_lun0 ............................................................................... [lun0 block/SANROOT (rw)]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0 ..................................... [block/SANROOT (/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0)]
  | |   o- portals .................................................................................................... [Portals: 4]
  | |     o- 172.16.20.40:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.41:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.42:3260 ................................................................................................ [OK]
  | |     o- 172.16.20.43:3260 ................................................................................................ [OK]
  | o- iqn.2003-01.org.linux-iscsi.debian.x8664 .......................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1993-08.org.debian:01:2972f6b5fc7 ................................................................ [Mapped LUNs: 2]
  |     |   o- mapped_lun0 .................................................................................. [lun0 block/SAN1 (rw)]
  |     |   o- mapped_lun1 .................................................................................. [lun1 block/SAN2 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 2]
  |     | o- lun0 .......................................................................................... [block/SAN1 (/dev/vdb)]
  |     | o- lun1 .......................................................................................... [block/SAN2 (/dev/vdc)]
  |     o- portals .................................................................................................... [Portals: 4]
  |       o- 172.16.20.40:3260 ................................................................................................ [OK]
  |       o- 172.16.20.41:3260 ................................................................................................ [OK]
  |       o- 172.16.20.42:3260 ................................................................................................ [OK]
  |       o- 172.16.20.43:3260 ................................................................................................ [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
/> 

Categories: 

Keywords: 

Like: 

24 November, 2016 09:17AM by Ritesh Raj Sarraf

SAN Updates for Debian Stretch

Now that we prepare for the next Debian Stable release (Stretch), it is time to provide some updates on what the current state of some of the (storage related) packages in Debian is. This is not an update on the complete list of packages related to storage, but it does cover some of them.

 

REMOVALS

  • iscsitarget - The iscsitarget stood as a great SCSI target for the Linux kernel. It seems to have had a good user base not just in Linux but also with VMWare users. But this storage target was always out-of-tree. With LIO having gotten merged as the default in-kernel SCSI Target, development on iscsitarget seems to have stalled. In Debian, for Stretch, there will be no iscsitarget. The package is already removed from Debian Testing and Debian Unstable, and nobody has volunteered to take over it.
  • system-storage-manager - This tool intended to be a simple unified storage tool, through which one could work with various storage technologies like LVM, BTRFS, cryptsetup, SCSI etc. But the upstream development hasn't really been much lately. For Debian Stable, it shouldn't be part of it, given it has some bugs.
  • libstoragemgmt - libstoragemgmt is a universal storage client-side library to talk to remote Storage Arrays. The project is active upstream. For Debian, the package is out-of-date and, now, also needs a maintainer. Unless someone picks up this package, it will not be part of Debian Stretch.

 

UPDATES

  • open-iscsi - This is the default iSCSI Initiator for Linux distributions. After a long slow development, upstream recently did a new release. This new release accomplished an important milestone; Hardware Offloading for QLogic cards. A special thanks to Frank Fegert, who helped with many aspects of the new iscsiuio package. And thanks to Christian Seiler, who is now co-maintaining the package, it is in great shape. We have fixed some long outstanding bugs and open-iscsi now has much much better integration with the whole system. For Jessie too, we have the up-to-date open-iscsi pacakges (including the new iscsiuio package, with iSCSI Offload) available through jessie-packports
  • open-isns - iSNS is the Naming Service for Storage. This is a new package in Debian Stretch. For users on Debian Jessie, Christian's efforts have made the open-isns package available in jessie-backports too.
  • multipath-tools - After years of slow development, multipath-tools too saw some active development this year, thanks to Xose and Christophe. The Debian version is up-to-date with the latest upstream release. For Debian Stretch, multipath-tools should have good integration with systemd.
  • sg3-utils - sg3 provides simple tools to query, using SCSI commands. The package is up-to-date and in good shape for Debian Stretch.
  • LIO Target - This is going to be the big entry for Debian Stretch. LIO is the in-kernel SCSI Target for Linux. For various reasons, we did not have LIO in Jessie. For Stretch, thanks to Christian Seiler and Christophe Vu-Brugier, we now have the well maintained -fb fork into Debian, which will replace the initial packages from the pre-fork upstream. The -fb fork is maintained by Andy Grover, and now, seems to have users from many other distributions and the kernel community. And given that LIO -fb branch is also part of the RHEL product family, we hope to see a well maintained project and an active upstream. The older packages: targetcli, python-rtslib and python-configshell shall be removed from the archive soon.

 

Debian users and derivatives, using these storage tools, may want to test/report now. Because once Stretch is released, getting new fixes in may not be easy enough. So please, if you have reliance on these tools, please test and report bugs, now.

Categories: 

Keywords: 

Like: 

24 November, 2016 08:51AM by Ritesh Raj Sarraf

Michael Stapelberg

Debian stretch on the Raspberry Pi 3

The last couple of days, I worked on getting Debian to run on the Raspberry Pi 3.

Thanks to the work of many talented people, the Linux kernel in version 4.8 is _almost_ ready to run on the Raspberry Pi 3. The only missing thing is the bcm2835 MMC driver, which is required to read the root file system from the SD card. I’ve asked our maintainers to include the patch for the time being.

Aside from the kernel, one also needs a working bootloader, hence I used Ubuntu’s linux-firmware-raspi2 package and uploaded the linux-firmware-raspi3 package to Debian. The package is currently in the NEW queue and needs to be accepted by ftp-master before entering Debian.

The most popular method of providing a Linux distribution for the Raspberry Pi is to provide an image that can be written to an SD card. I made two little changes to vmdebootstrap (#845439, #845526) which make it easier to create such an image.

The Debian wiki page https://wiki.debian.org/RaspberryPi3 describes the current state of affairs and should be updated, as this blog post will not be updated.

As a preview version (i.e. unofficial, unsupported, etc.) until all the necessary bits and pieces are in place to build images in a proper place in Debian, I built and uploaded the resulting image. Find it at https://people.debian.org/~stapelberg/raspberrypi3/. To install the image, insert the SD card into your computer (I’m assuming it’s available as /dev/sdb) and copy the image onto it:

$ wget https://people.debian.org/~stapelberg/raspberrypi3/2016-11-24-raspberry-pi-3-stretch-PREVIEW.img
$ sudo dd if=2016-11-24-raspberry-pi-3-stretch-PREVIEW.img of=/dev/sdb bs=5M

I hope this initial work on getting Debian booted will motivate other people to contribute little improvements here and there. A list of current limitations and potential improvements can be found on the RaspberryPi3 Debian wiki page.

24 November, 2016 07:45AM

November 23, 2016

hackergotchi for Joachim Breitner

Joachim Breitner

microG on Jolla

I am a incorrigibly in picking non-mainstream, open smartphones, and then struggling hard. Back then in 2008, I tried to use the OpenMoko FreeRunner, but eventually gave up because of hardware glitches and reverted to my good old Siemens S35. It was not that I would not be willing to put up with inconveniences, but as soon as it makes live more difficult for the people I communicate with, it becomes hard to sustain.

Two years ago I tried again, and got myself a Jolla phone, running Sailfish OS. Things are much nicer now: The hardware is mature, battery live is good, and the Android compatibility layer enables me to run many important apps that are hard to replace, especially the Deutsche Bahn Navigator and various messengers, namely Telegram, Facebook Messenger, Threema and GroupMe.

Some apps that require Google Play Services, which provides a bunch of common tasks and usually comes with the Google Play store would not run on my phone, as Google Play is not supported on Sailfish OS. So far, the most annoying ones of that sort were Uber and Lyft, making me pay for expensive taxis when others would ride cheaper, but I can live with that. I tried to install Google Play Services from shady sources, but it would regularly crash.

Signal on Jolla

Now in Philadelphia, people urged me to use the Signal messenger, and I was convinced by its support for good end-to-end crypto, while still supporting offline messages and allowing me to switch from my phone to my desktop and back during a conversation. The official Signal app uses Google Cloud Messaging (GCM, part of Google Play Services) to get push updates about new posts, and while I do not oppose this use of Google services (it really is just a ping without any metadata), this is a problem on Sailfish OS.

Luckily, the Signal client is open source, and someone created a “LibreSignal” edition that replaced the use of GCM with websockets, and indeed, this worked on my phone, and I could communicate.

Things were not ideal, though: I would often have to restart the app to get newly received messages; messages that I send via Signal Desktop would often not show up on the phone and, most severe, basically after every three messages, sending more messages from Desktop would stop working for my correspondents, which freaked them out. (Strangely it continued working from their phone app, so we coped for a while.)

So again, my choice of non-standard devices causes inconveniences to others. This, and the fact that the original authors of Signal and the maintainers of LibreSignal got into a fight that ended LibreSignal discontinued, meant that I have to change something about this situation. I was almost ready to give in and get myself a Samsung S7 or something boring of the sort, but then I decided to tackle this issue once more, following some of the more obscure instructions out there, trying to get vanilla Signal working on my phone. About a day later, I got it, and this is how I did it.

microG

So I need Google Play Services somehow, but installing the “real thing” did not seem to be very promising (I tried, and regularly got pop-ups telling me that Play Services has crashed.) But I found some references to a project called “microG”, which is an independent re-implementation of (some of) of the play services, in particular including GCM.

Installing microG itself was easy, as you can add their repository to F-Droid. I installed the core services, the services framework and the fake store apps. If this had been all that was to do, things would be easy!

Play Store detection work arounds

But Signal would still complain about the lack of Google Play Services. It asks Android if an app with a certain name is installed, and would refuse to work if this app does not exist. For some reason, the microG apps cannot just have the names of the “real” Google apps.

There seem to be two ways of working around this: Patching Signal, or enabling Signature Spoofing.

The initially most promising instructions (which are in a README in a tarball on a fishy file hoster linked from an answer on the Jolla support forum…) suggested patching Signal, and actually came both with a version of an app called “Lucky Patcher” as well as a patched Android package, but both about two years old. I tried a recent version of the Lucky Patcher, but it failed to patch the current version of Signal.

Signature Spoofing

So on to Signature Spoofing. This is a feature of some non-standard Android builds that allow apps (such as microG) to fake the existence of other apps (the Play Store), and is recommended by the microG project. Sailfish OS’s Android compatibility layer “Alien Dalvik” does not support it out of the box, but there is a tool “tingle” that adds this feature to existing Android systems. One just has to get the /system/framework/framework.jar file, put it into the input folder of this project, run python main.py, select 2, and copy the framework.jar from output/ back. Great.

Deodexing Alien Dalvik

Only that it only works on “deodexed” files. I did not know anything about odexed Android Java classes (and did not really want to know), but there was not way around. Following this explanation I gathered that one finds files foo.odex in the Android system folder, runs some tool on them to create a classes.dex file, and adds that to the corresponding foo.jar or foo.apk file, copies this back to the phone and deletes the foo.odex file.

The annoying this is that one does not only have to do it for framework.jar in order to please tingle, because if one does it to one odex file, one has to do to all! It seems that for people using Windows, the Universal Deodexer V5 seems to be a convenient tool, but I had to go more manually.

So I first fetched “smali”, compiled it using ./gradlew build. Then I fetched the folders /opt/alien/system/framework and /opt/alien/system/app from the phone (e.g. using scp). Keep a backup of these in case something breaks. Then I ran these commands (disclaimer: I fetched these from my bash history and slightly cleaned them up. This is not a fire-and-forget script! Use it when you know what it and you are doing):

cd framework
for file in *.odex
do
  java -jar ~/build/smali/baksmali/build/libs/baksmali.jar deodex $file -o out
  java -jar ~/build/smali/smali/build/libs/smali.jar a out -o classes.dex
  zip -u $(basename $file .odex).jar classes.dex
  rm -rf out classes.dex $file
done
cd ..

cd app
for file in *.odex
do
  java -jar ~/build/smali/baksmali/build/libs/baksmali.jar deodex -d ../framework $file -o out
  java -jar ~/build/smali/smali/build/libs/smali.jar a out -o classes.dex
  zip -u $(basename $file .odex).apk classes.dex
  rm -rf out classes.dex $file
done
cd ..

The resulting framework.jar can now be patched with tingle:

mv framework/framework.jar ~/build/tingle/input
cd ~/build/tingle
./main.py
# select 2
cd -
mv ~/build/tingle/output/framework.jar framework/framework.jar

Now I copy these framework and app folders back on my phone, and restart Dalvik:

devel-su systemctl restart aliendalvik.service

It might start a bit slower than usually, but eventually, all the Android apps should work as before.

The final bit that was missing in my case was that I had to reinstall Signal: If it is installed before microG is installed, it does not get permission to use GCM, and when it tries (while registering: After generating the keys) it just crashes. I copied /data/data/org.thoughtcrime.secretsms/ before removing Signal and moved it back after (with cp -a to preserve permissions) so that I could keep my history.

And now, it seems, vanilla Signal is working just fine on my Jolla phone!

What’s missing

Am I completely happy with Signal? No! An important feature that it is lacking is a way to get out all data (message history including media files) in a file format that can be read without Signal; e.g. YAML files or clean HTML code. I do want to be able to re-read some of the more interesting conversations when I am 74 or 75, and I doubt that there will be a Signal App, or even Android, then. I hope that this becomes available in time, maybe in the Desktop version.

I would also hope that pidgin gets support to the Signal protocol, so that I conveniently use one program for all my messaging needs on the desktop.

Finally it would be nice if my Signal identity was less tied to one phone number. I have a German and a US phone number, and would want to be reachable under both on all my clients. (If you want to contact me on Signal, use my US phone number.)

Alternatives

Could I have avoided this hassle by simply convincing people to use something other than Signal? Tricky, at the moment. Telegram (which works super reliable for me, and has a pidgin plugin) has dubious crypto and does not support crypto while using multiple clients. Threema has no desktop client that I know of. OTR on top of Jabber does not support offline messages. So nothing great seems to exist right now.

In the long run, the best bet seems to be OMEMO (which is, in essence, the Signal protocol) on top of Jabber. It is currently supported by one Android Jabber client (Conversations) and one Desktop application (gajim, via a plugin). I should keep an eye on pidgin support for OMEMO and other development around this.

23 November, 2016 05:44PM by Joachim Breitner (mail@joachim-breitner.de)

hackergotchi for Tanguy Ortolo

Tanguy Ortolo

Generate man pages for awscli

No man pages, but almost

The AWS Command Line Interface, which is available in Debian, provides no man page. Instead, that tool has an integrated help system, which allows you to run commands such as aws rds help, that, for what I have seen, generates some reStructuredText, then converts it to a man page in troff format, then calls troff to convert it to text with basic formatting, and eventually passes it to a pager. Since this is close to what man does, the result looks like a degraded man page, with some features missing such as the adaptation to the terminal width.

Well, this is better than nothing, and better than what many under-documented tools can offer, but for several reasons, it still sucks: most importantly, it does not respect administrators' habits and it does not integrate with the system man database. You it does not allow you to use commands such as apropos, and you will get no man page name auto-completion from your shell since there is no man page.

Generate the man pages

Now, since the integrated help system does generate a man page internally, we can hack it to output it, and save it to a file:

Description: Enable a mode to generate troff man pages
 The awscli help system internally uses man pages, but only to convert
 them to text and show them with the pager. This patch enables a mode
 that prints the troff code so the user can save the man page.
 .
 To use that mode, run the help commands with an environment variable
 OUTPUT set to 'troff', for instance:
     OUTPUT='troff' aws rds help
Forwarded: no
Author: Tanguy Ortolo <tanguy+debian@ortolo.eu>
Last-Update: 2016-11-22

Index: /usr/lib/python3/dist-packages/awscli/help.py
===================================================================
--- /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
+++ /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
@@ -49,6 +49,8 @@
     Return the appropriate HelpRenderer implementation for the
     current platform.
     """
+    if 'OUTPUT' in os.environ and os.environ['OUTPUT'] == 'troff':
+        return TroffHelpRenderer()
     if platform.system() == 'Windows':
         return WindowsHelpRenderer()
     else:
@@ -97,6 +99,15 @@
         return contents


+class TroffHelpRenderer(object):
+    """
+    Render help content as troff code.
+    """
+
+    def render(self, contents):
+        sys.stdout.buffer.write(publish_string(contents, writer=manpage.Writer()))
+
+
 class PosixHelpRenderer(PagingHelpRenderer):
     """
     Render help content on a Posix-like system.  This includes

This patch must be applied from the root directory with patch -p0, otherwise GNU patch will not accept to work on files with absolute names.

With that patch, you can run help commands with an environment variable OUTPUT='troff' to get the man page to use it as you like, for instance:

% OUTPUT='troff' aws rds help > aws_rds.1
% man -lt aws_rds.1 | lp

Generate all the man pages

Now that we are able to generate the man page of any aws command, all we need to generate all of them is a list of all the available commands. This is not that easy, because the commands are somehow derived from functions provided by a Python library named botocore, which are derived from a bunch of configuration files, and some of them are added, removed or renamed. Anyway, I have been able to write a Python script that does that, but it includes a static list of these modifications:

#! /usr/bin/python3

import subprocess
import awscli.clidriver


def write_manpage(command):
    manpage = open('%s.1' % '_'.join(command), 'w')
    command.append('help')
    process = subprocess.Popen(command,
            env={'OUTPUT': 'troff'},
            stdout=manpage)
    process.wait()
    manpage.close()


driver = awscli.clidriver.CLIDriver()
command_table = driver._get_command_table()

renamed_commands = \
    {
        'config': 'configservice',
        'codedeploy': 'deploy',
        's3': 's3api'
    }
added_commands = \
    {
        's3': ['cp', 'ls', 'mb', 'mv', 'presign', 'rb', 'rm', 'sync',
               'website']
    }
removed_subcommands = \
    {
        'ses': ['delete-verified-email-address',
                'list-verified-email-addresses',
                'verify-email-address'],
        'ec2': ['import-instance', 'import-volume'],
        'emr': ['run-job-flow', 'describe-job-flows',
                'add-job-flow-steps', 'terminate-job-flows',
                'list-bootstrap-actions', 'list-instance-groups',
                'set-termination-protection',
                'set-visible-to-all-users'],
        'rds': ['modify-option-group']
    }
added_subcommands = \
    {
        'rds': ['add-option-to-option-group',
                'remove-option-from-option-group']
    }

# Build a dictionary of real commands, including renames, additions and
# removals.
real_commands = {}
for command in command_table:
    subcommands = []
    subcommand_table = command_table[command]._get_command_table()
    for subcommand in subcommand_table:
        # Skip removed subcommands
        if command in removed_subcommands \
                and subcommand in removed_subcommands[command]:
            continue
        subcommands.append(subcommand)
    # Add added subcommands
    if command in added_subcommands:
        for subcommand in added_subcommands[command]:
            subcommands.append(subcommand)
    # Directly add non-renamed commands
    if command not in renamed_commands:
        real_commands[command] = subcommands
    # Add renamed commands
    else:
        real_commands[renamed_commands[command]] = subcommands
# Add added commands
for command in added_commands:
    real_commands[command] = added_commands[command]

# For each real command and subcommand, generate a manpage
write_manpage(['aws'])
for command in real_commands:
    write_manpage(['aws', command])
    for subcommand in real_commands[command]:
        write_manpage(['aws', command, subcommand])
                         'sync', 'website']}

This script will generate more than 2,000 man page files in the current directory; you will then be able to move them to /usr/local/share/man/man1.

Since this is a lot of man pages, it may be appropriate to concatenate them by major command, for instance all the aws rds together…

23 November, 2016 04:25PM by Tanguy

November 22, 2016

hackergotchi for Lars Wirzenius

Lars Wirzenius

Debian miniconf in Cambridge

I spent a few days in Cambridge for a minidebconf. This is a tiny version of the full annual Debconf. We had a couple of days for hacking, and another two days for talks.

I spent my hacking time on thinking about vmdebootstrap (my tool for generating disk images with an installed Debian), and came to the conclusion I need to atone my sins for writing such crappy code by rewriting it from scratch to be nicer to use. I gave a talk about this, too. The mailing list post has the important parts, and meetings-archive has a video.

I haven't started the rewrite, and it's not going to make it for stretch.

I also gave two other talks, on the early days of Linux, and Qvarn, the latter being what I do at work.

Thank you to ARM, for sponsoring the location, and the other sponsors for sponsoring food. These in-real-life meetings between developers are important for the productivity and social cohesion of Debian.

22 November, 2016 05:20PM

hackergotchi for Steve Langasek

Steve Langasek

A new chapter

I don't often write on this blog, and when I do, it's either tech related, or light life stuff.

Over the next few weeks, it's going to get a lot more political. If you currently follow this blog for its technical content, you may be tempted to tune out. I would encourage you to stay and listen. I'm passionate about the technology that I work on; but the greatest problems facing our world today are not ones that will be solved with software.

American democracy is in bad shape, and it's because of what we're doing to it. This is not a problem of the Right or of the Left; it is not a problem that began with the election of Donald Trump, and it's not a problem that will go away at the end of his term. It is partly a structural problem with the way our elections work, but more than that it's a problem of how we're splitting into separate tribes, isolating ourselves from those who don't agree with us.

As Russ Allbery wrote the morning after the election, everything about how we organize ourselves online today - and how we let Facebook and Twitter organize us - leads us to surround ourselves with people who already think the same way we do. That leaves all of us with huge blind spots for other people in our country, and it stifles the free exchange of ideas that is so essential for a healthy democracy. We need leaders who will work to make America a better and more just place for all our neighbors, not just a two-party system that plays tug-of-war using two different sets of voters that feel shut out. And the way we organize ourselves today (online and off) does not let us recognize those leaders.

There's a lot of talk now about Facebook changing how it decides what to show people; and maybe they can manage to help everyone's online experience be a little less of a bubble. But part of the change needs to come from us. We need to be willing to engage, civilly, with people whose perspective is different from ours, and make the effort to understand where the other is coming from.

So for the next few weeks, I'm going to talk. And I'm going to listen.

I have no unique qualifications to speak about the country's issues. But I do have a perspective of my own, which might be different enough from yours to be useful. I was born and raised in Iowa, and graduated from college there. This election cycle, I learned that Iowa holds the distinction of being the state with the lowest percentage of college-educated whites. I'm part of that statistic, because a few years after graduating I moved to Portland, Oregon - a place that's notoriously so far to the left of what we think of as the middle, that it actually has anarchists who would shamefully use a peaceful protest as cover to commit property crime. So I know a few things about the people in each state, that I think the other should hear.

I'm also that rarest of creatures, a Portlander who goes to church (Catholic). But I still choose as my neighbors the weird, wonderful, and welcoming community that we have here, whatever Glenn Beck might think.

I have a son, and I worry about what kind of world he'll grow up to live in. I work in software, which means I'm doing a lot better than a lot of people in the country right now; it also means that from where I sit, I see trends already in progress that will have an effect on the working class and the middle class that makes NAFTA look like a gnat's fart in comparison. And so I worry for what kind of world we will all live in, if we don't make some changes fast.

Let's have a conversation. No comments enabled on this blog, but you can find me on G+ or on Facebook.

22 November, 2016 07:06AM

November 21, 2016

Mike Gabriel

Please Welcome D0n1elT to the FLOSS World

TL;DR;

If you run a FLOSS development project and you notice D0n1elT appearing on your IRC channel, please give him a warm welcome. D0n1elT is a young man highy talented in various FLOSS related topics already. He probably needs some guidance at the beginning and I hope he won't be too shy to ask for it. But you can be sure: your channel has been joined by someone you should consider as a future resource.

The Long Story

During the last two weeks I had the great pleasure of supervising a fine young man (very young, still, indeed) in all sorts of IT topics. This young man turned out to be so skilled and interested in various FLOSS related areas, I really want to introduce him to all of you.

The young man's real name is Daniel Teichmann. On IRC he may appear under his nick: D0n1elT. His GnuPG Fingerprint is: 6C6E 7F8F F7E8 B22E FC76 E9F7 8A79 028F DA56 7C6C.

Daniel goes to a local school here in Nothern Germany, near where I live. He attends the 9th grade at his school, and as common for students of his age and grade, practical training was scheduled for the last two weeks.

Daniel had originally applied for practical training at some other business near his place of living (which is quite far off from the school, actually). However, that company cancelled his training position two work days before the training was supposed to start. Daniel's teacher rang me up and asked for help. He advertised Daniel as someone who is far advanced in IT topics compared to his co-students. "He even writes his own programs (in Java and C++)."

Spontaneously, Andreas Buchholz (CEO of LOGO EDV-Systeme GmbH) and I decided to accept Daniel as a trainee. Without having met him, with no application interview beforehand. The deal was: Daniel comes to Andreas business location in Kiel (40-50km away from Daniel's place of living) and I (working as freelancer for LOGO on a regular basis) do the supervising part.

On day one and two, as a warm-up, Daniel installed a Debian Edu Main Server, worked himself through GOsa, LDAP, SSH, GnuPG, Jabber and IRC and configured two routers. All topics were new to him and I could hardly think of new tasks to give to him. As means of communication we set up a Jabber account, then an IRC account (as backup). However, it turned out that Daniel really got a hang of IRC over the next couple of days, so we used that as primary communication channel.

Daniel had already programmed various projects in Java (whereas I have never touched Java, so far :-( ). He has written plugins for Minecraft servers. He knows well how to implement object oriented coding models. His coding style looks very good and clean (esp. for someone who has never head a nitpicking code reviewer). He started coding at the age of 9.

Instead of diving into Java (where I would not have been of much help, anyway) I decided to provide him with some really basic and Unix-like knowledge: Bash scripting. I wanted to see how he handles another "language" and how he applies his Java knowledge to a lower level, syntactically weaker language. Guess what, he managed that assignment very well.

Working on Impressive Display

At Daniel's school we run substitute teacher info screens based on a fancy Bash script, named impressive-display, and the impressive PDF viewer. The Impressive Display tool is available in Debian testing/unstable under the same name.

So over the next couple of days we worked on Impressive Display. Daniel contributed so many new code passages, conceptual ideas and security concerns, that I decided to make him co-copyright holder. Every change contributed by him received intensive testing before committing to Git. While working on Impressive Display, collaborating with Daniel via Git was a mere pleasure. In his spare time Daniel likes watching Github tutorials. Quite extraordinary.

The result is a new major release of Impressive Display: Version 0.3.1 (bumped up from 0.2.3). We added the feature of handling info screen farms based on PXE boot images. It is now possible to configure as many different info screens as needed within the same PXE bootable chroot.

Furthermore, Impressive Display now has a PDF presentation (written in LaTeX Beamer) that documents how to setup your own info screens. The PDF presentation is the default PDF that comes up when you start Impressive Display directly after installation.

Investigating other Realms

We also took a deeper look at remote desktop stuff, one of my most favourite topics. By that impulse Daniel set up his first Vserver machine at some hosting provider. He figured out how to run X2Go Server on that machine with an XFCE desktop. Next step was to run the irssi instance from his notebook inside a screen session on the Vserver. Some days later, Daniel PM'ed me: "I have an IRC bouncer now...".

Quintessence

It was a great pleasure meeting this young, highly curious and already highly skilled young man over the past two weeks.

Daniel, it was an asset to me working with you. You are such a fast learner when it comes to getting accustomed to new working environments, it is amazing. I cannot deny having observed the tendency of preferring rather geeky tools. I was highly delighted, that What's-That and Facebook are nothing that rocks you so much. Unfortunately, all of the above makes you quite unique and non-mainstream among people of your age.

My wish for you (and the FLOSS world) is that you start getting in touch with other (FLOSS) developers, maybe of your age, maybe older, and that you (if this is what you want) become an asset to the world of Free Software. The Free Software world can be a world where technical, political and spiritual work become one with friendship among people.

Take care and farewell! I am sure, we will meet again.

light+love Mike Gabriel (aka sunweaver on IRC and debian.org)

21 November, 2016 10:32PM by sunweaver

hackergotchi for Daniel Pocock

Daniel Pocock

21 November 1916

There has been a lot of news recently about the 100th anniversaries of various events that took place during the Great War.

On 21 November 1916, the SS Hunscraft sailed from Southampton to France. My great grandfather, Robert Pocock, was aboard.

He was part of the Australian Imperial Force, 3rd Divisional Train.

It's sad that Australians had to travel half way around the world to break up fist fights and tank battles. Sadder still that some people who romanticize the mistakes of imperialism are being appointment to significant positions of power.

Fortunately my great grandfather returned to Australia in one piece, many Australians didn't.

Robert Pocock's war medals

21 November, 2016 06:31PM by Daniel.Pocock

hackergotchi for Gustavo Noronha Silva

Gustavo Noronha Silva

A tale of cylinders and shadows

Like I wrote before, we at Collabora have been working on improving WebKitGTK+ performance for customer projects, such as Apertis. We took the opportunity brought by recent improvements to WebKitGTK+ and GTK+ itself to make the final leg of drawing contents to screen as efficient as possible. And then we went on investigating why so much CPU was still being used in some of our test cases.

The first weird thing we noticed is performance was actually degraded on Wayland compared to running under X11. After some investigation we found a lot of time was being spent inside GTK+, painting the window’s background.

Here’s the thing: the problem only showed under Wayland because in that case GTK+ is responsible for painting the window decorations, whereas in the X11 case the window manager does it. That means all of that expensive blurring and rendering of shadows fell on GTK+’s lap.

During the web engines hackfest, a couple of months ago, I delved deeper into the problem and noticed, with Carlos Garcia’s help, that it was even worse when HiDPI displays were thrown into the mix. The scaling made things unbearably slower.

You might also be wondering why would painting of window decorations be such a problem, anyway? They should only be repainted when a window changes size or state anyway, which should be pretty rare, right? Right, that is one of the reasons why we had to make it fast, though: the resizing experience was pretty terrible. But we’ll get back to that later.

So I dug into that, made a few tries at understanding the issue and came up with a patch showing how applying the blur was being way too expensive. After a bit of discussion with our own Pekka Paalanen and Benjamin Otte we found the root cause: a fast path was not being hit by pixman due to the difference in scale factors on the shadow mask and the target surface. We made the shadow mask scale the same as the surface’s and voilà, sane performance.

I keep talking about this being a performance problem, but how bad was it? In the following video you can see how huge the impact in performance of this problem was on my very recent laptop with a HiDPI display. The video starts with an Epiphany window running with a patched GTK+ showing a nice demo the WebKit folks cooked for CSS animations and 3D transforms.

After a few seconds I quickly alt-tab to the version running with unpatched GTK+ – I made the window the exact size and position of the other one, so that it is under the same conditions and the difference can be seen more easily. It is massive.

Yes, all of that slow down was caused by repainting window shadows! OK, so that solved the problem for HiDPI displays, made resizing saner, great! But why is GTK+ repainting the window even if only the contents are changing, anyway? Well, that turned out to be an off-by-one bug in the code that checks whether the invalidated area includes part of the window decorations.

If the area being changed spanned the whole window width, say, it would always cause the shadows to be repainted. By fixing that, we now avoid all of the shadow drawing code when we are running full-window animations such as the CSS poster circle or gtk3-demo’s pixbufs demo.

As you can see in the video below, the gtk3-demo running with the patched GTK+ (the one on the right) is using a lot less CPU and has smoother animation than the one running with the unpatched GTK+ (left).

Pretty much all of the overhead caused by window decorations is gone in the patched version. It is still using quite a bit of CPU to animate those pixbufs, though, so some work still remains. Also, the overhead added to integrate cairo and GL rendering in GTK+ is pretty significant in the WebKitGTK+ CSS animation case. Hopefully that’ll get much better from GTK+ 4 onwards.

21 November, 2016 05:04PM by kov

Reproducible builds folks

Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016:

Media coverage

Elsewhere in Debian

  • dpkg 1.18.14 has migrated to stretch.

  • Chris Lamb filed #844431 ("packages should build reproducibly") against debian-policy.

  • Ximin worked on glibc reproducibility this week, catching some bugs in disorderfs, FUSE, as well as glibc itself.

Documentation update

Packages reviewed and fixed, and bugs filed

Reviews of unreproducible packages

43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

4 issue types have been added:

Weekly QA work

During our reproducibility testing, some FTBFS bugs have been detected and reported by:

  • Chris Lamb (26)
  • Daniel Stender (1)
  • Filip Pytloun (1)
  • Lucas Nussbaum (28)
  • Michael Biebl (1)

strip-nondeterminism development

disorderfs development

  • #844498 ("disorderfs: using it for building kills the host")

debrebuild development

debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild").

debrepatch development

debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible.

debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts.

reprotest development

Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before.

He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process.

These changes will be tested some more and then released in the near future as reprotest 0.4.

tests.reproducible-builds.org

  • Debian:

    • An index of our usertagged bugs page was added by Holger after a Q+A session in Cambridge.
    • Holger also setup two new i386 builders, build12+16, for >50% increased build performance. For this, we went from 18+17 cores on two 48GB machines to 10+10+9+9 cores on four 36GB ram machines… and from 16 to 24 builder jobs. Thanks to Profitbricks for providing us with all these ressources once more!
    • h01ger also tried to enable disorderfs again, but hit #844498, which brought down the i386 builders, so he disabled it again. Next will be trying disorderfs on armhf or amd64, to see whether this bug also manifests there.

Misc.

This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

21 November, 2016 12:47PM

Arturo Borrero González

Great Debian meeting in Seville

Debian meeting Seville

Last week we had an interesting Debian meeting in Seville, Spain. This has been the third time (in recent years) the local community meets around Debian.

We met at about 20:00 at Rompemoldes, a crafts creation space. There we had a very nice dinner while talking about Debian and FLOSS. The dinner was sponsored by the Plan4D assosiation.

The event was joined by almost 20 people which different relations to Debian:

  • Debian users
  • DDs
  • Debian contributors
  • General FLOSS interested people

I would like to thank all the attendants and Pablo Neira from Plan4D for the organization.

I had to leave the event after 3.5 hours of great talking and networking, but the rest of the people stayed there. The climate was really good :-)

Looking forward to another meeting in upcomings times!

Header picture by Ana Rey.

21 November, 2016 05:00AM

hackergotchi for Steve Kemp

Steve Kemp

Detecting fraudulent signups?

I run a couple of different sites that allow users to sign-up and use various services. In each of these sites I have some minimal rules in place to detect bad signups, but these are a little ad hoc, because the nature of "badness" varies on a per-site basis.

I've worked in a couple of places where there are in-house tests of bad signups, and these usually boil down to some naive, and overly-broad, rules:

  • Does the phone numbers' (international) prefix match the country of the user?
  • Does the postal address supplied even exist?

Some places penalise users based upon location too:

  • Does the IP address the user submitted from come from TOR?
  • Does the geo-IP country match the users' stated location?
  • Is the email address provided by a "free" provider?

At the moment I've got a simple HTTP-server which receives a JSON post of a new users' details, and returns "200 OK" or "403 Forbidden" based on some very very simple critereon. This is modeled on the spam detection service for blog-comments server I use - something that is itself becoming less useful over time. (Perhaps time to kill that? A decision for another day.)

Unfortunately this whole approach is very reactive, as it takes human eyeballs to detect new classes of problems. Code can't guess in advance that it should block usernames which could collide with official ones, for example allowing a username of "admin", "help", or "support".

I'm certain that these systems have been written a thousand times, as I've seen at least five such systems, and they're all very similar. The biggest flaw in all these systems is that they try to classify users in advance of them doing anything. We're trying to say "Block users who will use stolen credit cards", or "Block users who'll submit spam", by correlating that behaviour with other things. In an ideal world you'd judge users only by the actions they take, not how they signed up. And yet .. it is better than nothing.

For the moment I'm continuing to try to make the best of things, at least by centralising the rules for myself I cut down on duplicate code. I'll pretend I'm being cool, modern, and sexy, and call this a micro-service! (Ignore the lack of containers for the moment!)

21 November, 2016 03:45AM

November 20, 2016

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Nageru documentation

Even though the World Chess Championship takes up a lot of time these days, I've still found some time for Nageru, my live video mixer. But this time it doesn't come in form of code; rather, I've spent my time writing documentation.

I spent some time fretting over what technical solution I wanted. I explicitly wanted end-user documentation, not developer documentation—I rarely find HTML-rendered versions of every member function in a class the best way to understand a program anyway. Actually, on the contrary: Having all sorts of syntax interwoven in class comments tends to be more distracting than anything else.

Eventually I settled on Sphinx, not because I found it fantastic (in particular, ReST is a pain with its bizarre variable punctuation-based syntax), but because I'm convinced it has all the momentum right now. Just like git did back in the day, the fact that the Linux kernel has chosen it means it will inevitably grow a quite large ecosystem, and I won't be ending up having to maintain it anytime soon.

I tried finding a balance between spending time on installation/setup (only really useful for first-time users, and even then, only a subset of them), concept documentation (how to deal with live video in general, and how Nageru fits into a larger ecosystem of software and equipment) and more concrete documentation of all the various features and quirks of Nageru itself. Hopefully, most people will find at least something that's not already obvious to them, without drowning in detail.

You can read the documentation at https://nageru.sesse.net/doc/, or if you want to send patches, the right place to patch is the git repository.

20 November, 2016 09:45PM

November 19, 2016

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

BH 1.62.0-1

The BH package on CRAN was updated to version 1.62.0. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages.

This release upgrades the version of Boost to the upstream version Boost 1.62.0, and adds three new libraries as shown in the brief summary of changes from the NEWS file which follows below.

Special thanks to Kurt Hornik and Duncan Murdoch for help tracking down one abort() call which was seeping into R package builds, and then (re-)testing the proposed fix. We are now modifying one more file ever so slightly to use ::Rf_error(...) instead.

Changes in version 1.62.0-1 (2016-11-15)

  • Upgraded to Boost 1.62 installed directly from upstream source

  • Added Boost property_tree as requested in #29 by Aydin Demircioglu

  • Added Boost scope_exit as requested in #30 by Kirill Mueller

  • Added Boost atomic which we had informally added since 1.58.0

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 November, 2016 01:07PM

hackergotchi for Keith Packard

Keith Packard

AltOS-Lisp-2

Updates to Altos Lisp

I wrote a few days ago about a tiny lisp interpreter I wrote for AltOS

Really, it's almost "done" now, I just wanted to make a few improvements

Incremental Collection

I was on a walk on Wednesday when I figured out that I didn't need to do a full collection every time; a partial collection that only scanned the upper portion of memory would often find plenty of free space to keep working for a while.

To recap, the heap is in two pieces; the ROM piece and the RAM piece. The ROM piece is generated during the build process and never changes afterwards (hence the name), so the only piece which is collected is the RAM piece. Collection works like:

chunk_low = heap base
new_top = heap base

For all of the heap
    Find the first 64 live objects above chunk_low
    Compact them all to new_top
    Rewrite references in the whole heap for them
    Set new_top above the new locations
    Set chunk_low above the old locations

top = new_top

The trick is to realize that there's really no need to start at the bottom of the heap; you can start anywhere you like and compact stuff, possibly leaving holes below that location in the heap. As the heap tends to have long-lived objects slowly sift down to the beginning, it's useful to compact objects higher than that, skipping the compaction process for the more stable area in memory.

Each time the whole heap is scanned, the top location is recorded. After that, incremental collects happen starting at that location, and when that doesn't produce enough free space, a full collect is done.

The collector now runs a bunch faster on average now.

Binary Searches

I stuck some linear searches in a few places in the code, the first was in the collector when looking to see where an object had moved to. As there are 64 entries, the search is reduced from 32 to 6 compares on average. The second place was in the frame objects, which hold the list of atom/value bindings for each lexical scope (including the global scope). These aren't terribly large, but a binary search is still a fine plan. I wanted to write down here the basic pattern I'm using for binary searches these days, which avoids some of the boundary conditions I've managed to generate in the past:

int find (needle) {
    int l = 0;
    int r = count - 1;
    while (l <= r) {
        int m = (l + r) >> 1;
        if (haystack[m] < needle)
            l = m + 1;
        else
            r = m - 1;
    }
    return l;
}

With this version, the caller can then check to see if there's an exact match, and if not, then the returned value is the location in the array where the value should be inserted. If the needle is known to not be in the haystack, and if the haystack is large enough to accept the new value:

void insert(needle) {
    int l = find(needle);

    memmove(&haystack[l+1],
        &haystack[l],
        (num - l) * sizeof (haystack[0]));

    haystack[l] = needle;
}

Similarly, if the caller just wants to know if the value is in the array:

bool exists(needle) {
    int l = find(needle);

    return (l < count && haystack[l] == needle);
}

Call with Current Continuation

Because the execution stack is managed on the heap, it's completely trivial to provide the scheme-like call with current continuation, which constructs an object which can be 'called' to transfer control to a saved location:

> (+ "hello " (call/cc (lambda (return) (setq boo return) (return "foo "))) "world")
"hello foo world"
> (boo "bar ")
"hello bar world"
> (boo "yikes ")
"hello yikes world"

One thing I'd done previously is dump the entire state of the interpreter on any error, and that included a full stack trace. I adopted that code for printing of these continuation objects:

boo
[
    expr:   (call/cc (lambda (return) (set (quote boo) return) (return "foo ")))
    state:  val
    values: (call/cc
             [recurse...]
             )
    sexprs: ()
    frame:  {}
]
[
    expr:   (+ "hello " (call/cc (lambda (return) (set (quote boo) return) (return "foo "))) "world")
    state:  formal
    values: (+
             "hello "
             )
    sexprs: ((call/cc (lambda (return) (set (quote boo) return) (return "foo ")))
             "world"
             )
    frame:  {}
]

The top stack frame is about to return from the call/cc spot with a value; supply a value to 'boo' and that's where you start. The next frame is in the middle of computing formals for the + s-expression. It's found the + function, and the "hello " string and has yet to get the value from call/cc or the value of the "world" string. Once the call/cc "returns", that value will get moved to the values list and the sexpr list will move forward one spot to compute the "world" value.

Implementing this whole mechanism took only a few dozen lines of code as the existing stack contexts were already a continuation in effect. The hardest piece was figuring out that I needed to copy the entire stack each time the continuation was created or executed as it is effectively destroyed in the process of evaluation.

I haven't implemented dynamic-wind yet; when I did that for nickle, it was a bit of a pain threading execution through the unwind paths.

Re-using Frames

I decided to try and re-use frames (those objects which hold atom/value bindings for each lexical scope). It wasn't that hard; the only trick was to mark frames which have been referenced from elsewhere as not-for-reuse and then avoid sticking those in the re-use queue. This reduced allocations even further so that for simple looping or tail-calling code, the allocator may never end up being called.

How Big Is It?

I've managed to squeeze the interpreter and all of the rest of the AltOS system into 25kB of Cortex-M0 code. That leaves space for the 4kB boot loader and 3kB of flash to save/restore the 3kB heap across resets.

Adding builtins to control timers and GPIOs would make this a reasonable software load for an Arduino; offering a rather different programming model for those with a taste for adventure. Modern ARM-based Arduino boards have plenty of flash and ram for this. It might be interesting to get this running on the Arduino Zero; there's no real reason to replace the OS either; porting the lisp interpreter into the bare Arduino environment wouldn't take long.

19 November, 2016 09:20AM

November 18, 2016

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

Rcpp 0.12.8: And more goodies

Yesterday the eighth update in the 0.12.* series of Rcpp made it to the CRAN network for GNU R where the Windows binary has by now been generated too; the Debian package is on its way as well. This 0.12.8 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, and the 0.12.7 release in September --- making it the twelveth release at the steady bi-montly release frequency. While we are keeping with the pattern, we have managed to include quite a lot of nice stuff in this release. None of it is a major feauture, though, and so we have not increased the middle number.

Among the changes this release are (once again) much improved exception handling (thanks chiefly to Jim Hester), better large vector support (by Qiang), a number of Sugar extensions (mostly Nathan, Qiang and Dan) and beginnings of new DateVector and DatetimeVectors classes, and other changes detailed below. We plan to properly phase in the new date(time) classes. For now, you have to use a #define such as this one in Rcpp.h which remains commented-out for now. We plan to switch this on as the new default no earlier than twelve months from now.

Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 843 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by eightyfour packages, or a full ten percent, just since the last release in early September!

Again, we are lucky to have such a large group of contributors. Among them, we have invited Nathan Russell to the Rcpp Core team given his consistently excellent pull requests (as well as many outstanding Stackoverflow answers for Rcpp). More details on changes are below.

Changes in Rcpp version 0.12.8 (2016-11-16)

  • Changes in Rcpp API:

    • String and vector elements now use extended R_xlen_t indices (Qiang in PR #560)

    • Hashing functions now return unsigned int (Qiang in PR #561)

    • Added static methods eye(), ones(), and zeros() for select matrix types (Nathan Russell in PR #569)

    • The exception call stack is again correctly reported; print methods and tests added too (Jim Hester in PR #582 fixing #579)

    • Variatic macros no longer use a GNU extensions (Nathan in PR #575)

    • Hash index functions were standardized on returning unsigned integers (Also PR #575)

  • Changes in Rcpp Sugar:

    • Added new Sugar functions rowSums(), colSums(), rowMeans(), colMeans() (PR #551 by Nathan Russell fixing #549)

    • Range Sugar now used R_xlen_t type for start/end (PR #568 by Qiang Kou)

    • Defining RCPP_NO_SUGAR no longer breaks the build. (PR #585 by Daniel C. Dillon)

  • Changes in Rcpp unit tests

    • A test for expression vectors was corrected.

    • The constructor test for datetime vectors reflects the new classes which treats Inf correctly (and still as a non-finite value)

  • Changes in Rcpp Attributes

    • An 'empty' return was corrected (PR #589 fixing issue #588, and with thanks to Duncan Murdoch for the heads-up)
  • Updated Date and Datetime vector classes:

    • The DateVector and DatetimeVector classes were renamed with a prefix old; they are currently typedef'ed to the existing name (#557)

    • New variants newDateVector and newDatetimeVector were added based on NumericVector (also #557, #577, #581, #587)

    • By defining RCPP_NEW_DATE_DATETIME_VECTORS the new classes can activated. We intend to make the new classes the default no sooner than twelve months from this release.

    • The capabilities() function can also be used for presence of this feature

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 November, 2016 11:41AM

November 17, 2016

Reproducible builds folks

Reproducible Builds: week 81 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 6 and Saturday November 12 2016:

Media coverage

Matthew Garrett blogged about Tor, TPMs and service integrity attestation and how reproducible builds are the base for systems integrity.

The Linux Foundation announced renewed funding for us as part of the Core Infrastructure Initiative. Thank you!

Outreachy updates

Maria Glukhova has been accepted into the Outreachy winter internship and will work with us the Debian reproducible builds team.

To quote her words

siamezzze: I've been accepted to #outreachy winter internship - going to
work with Debian reproducible builds team. So excited about that! <3
Debian

Toolchain development and fixes

dpkg:

  • Thanks to a series of dpkg uploads by Guillem Jover, all our toolchain changes are now finally available in sid!
  • This means your packages should now be reproducible without having to use our custom APT repository.
  • Ximin Luo opened #843925 to remind the fact that dpkg-buildpackage should sign buildinfo files.
  • We hope to have detailed post about the new dpkg and the new .buildinfo files for debian-devel-announce soon!

debrebuild:

  • srebuild / debrebuild work was resumed by Johannes Schauer and others in #774415.

Bugs filed

Chris Lamb:

Daniel Shahaf:

Niko Tyni:

Reiner Herrman:

Reviews of unreproducible packages

136 package reviews have been added, 5 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues.

3 issue types have been updated:

Weekly QA work

During of reproducibility testing, some FTBFS bugs have been detected and reported by:

  • Chris Lamb (29)
  • Niko Tyni (1)

diffoscope development

A new version of diffoscope 62~bpo8+1 was uploaded to jessie-backports by Mattia Rizzolo.

Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n2) lookup which was causing diffs of large packages such as GCC and glibc to take many more hours than was necessary. When this commit is released, we should hopefully see full diffs for such packages again. Currently we have 197 source packages which - when built - diffoscope fails to analyse.

buildinfo.debian.net development

  • Submissions with duplicate Installed-Build-Depends entries are rejected now that a bug in dpkg causing them has been fixed. Thanks to Guillem Jover.
  • Add a new page for every (source, version) combination, for example diffoscope 62.
  • DigitalOcean have generously offered to sponsor the hardware buildinfo.debian.net is running on.

tests.reproducible-builds.org

Debian:

  • For privacy reasons, the new dpkg-genbuildinfo includes Build-Path only if it is under /build. HW42 updated our jobs so this is the case for our builds too, so you can see the build path in the .buildinfo files.
  • HW42 also updated our jobs to vary the basename of the source extraction directory. This detects packages that incorrectly assume a $pkg-$version directory naming scheme (which is what dpkg-source -x gives but is not mandated by Debian nor always-true) or that they're being built from a SCM.
  • The new dpkg-genbuildinfo also records a sanitised Environment. This is different in our builds, so HW42, Reiner and Holger updated our jobs to hide these differences from diffoscope output.
  • Package-set improvements:
  • Valerie Young contributed four patches for our long-planned transition from SQLite to PostgreSQL.
  • In anticipation of the freeze, already-tested packages from unstable and testing on amd64 are now scheduled with equal priority.

reproducible-builds.org website

F-Droid was finally added to our list of partner projects. (This was an oversight and they had already been working with us for some time.)

Misc.

This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

17 November, 2016 07:46PM