Write-ups

TL;DR: Create a folder named "2016" in your "Desktop" or "Documents" (wherever you keep your work files). Move everything there. If you need anything again, take it back out.

* * *

There are many ways of organizing your folders in a directory structure. None of them will ever be complete satisfactory, because we use multiple dimensions to find our stuff. Sometimes we would like to look for a particular typology of project, or by the name of a collaborator, or by date.

Of all the dimensions by which you could organize a directory structure, what I do is to organize them by aspects of life (for most that will be "Work" and "Personal") and inside each area, by year.

Organizing your folders by year is not by itself better than other criteria of organizing things, but has two distinct advantages: it makes it easy to tidy up things, moving unused stuff out of the way, and it is backup-friendly.

A simple method, though not the one I follow which I explain below, is to keep only what you're currently using in the top level, and move stuff from previous years into one folder per year:

1. Create a folder with the name of the year that passed.
2. Move everything into that folder.
3. If you need any of that again, move it out of the folder and back to the top-level of your "Desktop" or "Documents."

* * *

The method I use is the following. I keep top-level folders "inv" for research, and "pers" for personal (in the past, "inv" was "research," which was too long; even before, it was "work," which didn't feel right to me).

Inside each of these top-level folders, I keep one folder per project, like this:

1. 2016_bigcrisisdata/
2. 2016_swdm/
3. 2016_digital_health/
4. 2016_petitions_modeling/
5. …
6. 2016_bureaucracy/
7. 2016_reviews/
8. 2016_recommendations/
9. Archive/

If I continue adding or modifying files in a folder, because I continue active on that project, I rename the folder, e.g., "2016_digital_health -> 2017_digital_health."

If I continue using some files as reference in a read-only manner, I keep those files with the year they have in the top-level folder.

If I stop using a folder, I move it to the “Archive.” From there, I copy or move stuff to an external backup when it gets several months old.

I keep no files in the top level, only folders. If I need to start anything, I create a top-level folder, and then move to the “Archive/” if it goes nowhere. Alternatively, I use some one of the generic folders: “{Year}_bureaucracy/”, “{Year}_reviews/”, “{Year}_recommendations/” are the generic ones I use now.

* * *

I backup continuously and automatically my working directory, as I try to keep there only the active projects and the recently archived ones. If I need to answer a question about something I did years ago (which doesn't happen often, but it happens), I go to to the external backup.

Over the years, I have found this way of organizing things makes things easy for me. In general, having a simple method for organizing your stuff can save you a lot of time and effort. Just choose a way and follow it, and make exceptions when necessary, in whatever way works best for you.

Happy New Year!

On April 25, 2016, the Cryptome foundation disclosed that a large zip archive containing what seemed to be private data from hundreds of thousands of customers of the Qatar National Bank (QNB, the largest bank in the Arab Gulf in terms of assets) had been posted online.

The archive

The archive (510MB compressed, 1.4GB uncompressed) contained about 15K files in many directories. The most obviously significant were (1) a set of database tables, and (2) a directory named "Folders."

The directory named "Folders" is a set of small dossiers of about 100 "notable" people in Qatar. It includes several sub-folders named "Al Jazeera" (the name of the Qatar-based news network), "Al Thani" (the name of the ruling family), "Police, security" and so on. Inside each folder, there is one or more files containing a mixture of data extracted from the QNB database (such as account number, passport number, and so on), and information from other sources, that varies from links to online profiles to a photo of the person, usernames and passwords. I say "notable" in quotes because the classification of people into folders here is a little bit dubious, for instance with some people named as spies when they are unlikely to be so.

The database tables contain profile information of about 300K-400K customers. This includes name, nationality, passport number, national ID number in Qatar, e-mail address, and physical address. It also includes card numbers, expiration dates, and account numbers for about 800K-900K debit and credit cards. There is some duplicate information between a set of main files and a set of back-ups, so the number varies depending on which files you look at.

My involvement

I worked in Qatar from mid-2012 to mid-2015 as a scientist in a national research institute. From my time in Qatar I knew that QNB is the first choice for many foreign workers, so the data leak had probably affected many of my colleagues who still work in Qatar.

After downloading the archive I did a quick search for e-mail addresses in my former institute, I found 4 people I knew, including their full names, which was a strong signal that the archive was authentic. Additionally, there were things that you only see in Qatar (such as writing "QATAR FOUNDASHON" instead of "QATAR FOUNDATION"). I alerted my colleagues, but didn't want to look more into the archive; I was not interested in learning anything private from them, and given I did not find my own e-mail address, I assumed this was old data from before I went to Qatar.

However, hours later and after being alerted by a friend, I did find my account in the archive. My e-mail ID was not included in the main files, but in one of the backups, and my national ID number in Qatar was in multiple places.

I looked more carefully in the archive and found the names of people I knew, but not their e-mail addresses. Then, I decided to create a tool to let people search for themselves if they were present in the archive or not.

The "QNB Verifier"

The tool I set up on April 26th was very simple. First, I hashed all the e-mails and Qatar national ID numbers I could found and uploaded the hashed identifiers to my web server. Second, I wrote a small Python script (my first program in Python) to receive a hashed user input and compare it against the stored hashes. Third, I used an Javascript implementation of MD5 to hash the input in the browser of the user.

I set-up a page to give access to this tool to users, and warned them that their hashed input could be observed while in transit, but indicated that no plain-text personal data would leave their browsers. I also made it clear I was not hosting the leaked files. I kept no logs of the user input but my service providers has statistics of how many people accessed the endpoint of the service. About 10,000 did so during the 5 days or so that the service was active.

In the meantime, QNB continued calling this a "social media speculation," a line they maintained for several days.

The denial-of-service attack

From April 27th onwards I started to get thousands of requests per second from a few IP addresses. This increased substantially the CPU usage of my account. My hosting provider froze all my websites (i.e. started serving static versions of them) except for this verification service (!) and warned me that if this high CPU usage continued they will suspend my account.

I added the IP addresses of the attacker to a blacklist but s/he kept on changing them. I spent hours playing whack-a-mole but fortunately for me the attack was not very sophisticated, and eventually my attacker gave up and stopped changing IP addresses. That brought my traffic down to normality and my websites were unfrozen.

The legal threats

On April 30th the service was featured in Doha News. Doha News is an independent online news service that operates from Doha. They knew me from my time in Qatar and had covered some of the work on predictive news analytics that my team did for Al Jazeera. They linked to the verification service and interviewed me over Twitter to know about my reasons and how this worked. The International Business Times also followed up on this story.

On the same day, I received an e-mail from Hispasec, a Spanish IT security firm. In their e-mail, they wrote in Spanish:

I am .... from Hispasec Systems, an international IT security company, in legal representation of QNB Bank from Qatar.

The entity had reported your 'qnb-ver' service for us to proceed to shut it down, because they understand it can be being utilized to collect e-mail addresses or bank account numbers:

URL: http://chato.cl/2016/qnb-ver/

We understand this is a special situation, but due to its importance and the requirements of our customer, we must emphasize that you must disable this content from your website.

Many thanks for your collaboration.

I responded a couple of days later and had some exchanges with them. I basically stated that I was not hosting the leaked files and that I was not collecting any data. They reinstated their request and warned me about the importance of promptly removing my service to "avoid to face any legal action concerning the spreading of the public damage and defamation." I have to say I never felt this was something to be concerned about; in Spain you are fairly safe unless you speak against the king.

Censorship

The Internet has always been heavily censored in Qatar, and many different content categories are not accessible from the country. Censorship is, however, quite easy to circumvent with VPNs, and many people use them.

On 2014 or 2015 they set-up a censorship page, that you can see in censor.qa that shows a cartoon and explains that you have accessed a page that contains prohibited materials. This makes censorship more "friendly," I guess.

My page was added to Qatar's blacklist on May 1st, 2016. I set-up an alternative, https-based access on the same day and publicized it, people continued being able to access the https-based service.

Incidentally, on May 1st, 2016, Qatar National Bank issued a statement indicating that the leak had only affected a "a portion of Qatar-based QNB customers." I would say that "portion" is very high, close to 100%. I did not hear about a single QNB customer that used my service and did not find his/her national ID or e-mail in the leaked files.

Downfall

On May 2nd QNB contacted my hosting provider to report a "phishing" attempt from my account. Phishing is when scammers send people e-mails asking them to follow a link that looks like a bank's homepage (or other institution), so that the victims enter their credentials in the fake site, and the scammers can steal them.

In response to the phishing complaint, and without any examination, my hosting provider proceeded to close all the websites hosted in my account, including my personal site, my research site, the website of my upcoming book, the personal website of my wife, an environmental portal that she maintains, and others.

I replied to the report with a detailed explanation that no personal data was being hosted, and no data was being stored, moreover that the service was designed to make this impossible by hashing the personal data on the users' browser. I received an automated response that pointed out the offending files were still there. Basically, a human would not look at the ban until the files were removed.

So, I had to shut down the service, and it took me about one day to get my hosting provider to bring the websites back into operation. I did not want to fight this. Their business is to have customers that don't require any attention, and if you use too much time from the support desk, they can easily revoke your contract and send you with a nice backup in search for another hosting. In my past experience that is 2-3 days of work that I don't have.

What did I learn?

In my opinion, data leaks should be regulated similarly to work place accidents. Companies should have to file an official report indicating what happened and who was affected. Most importantly, customers must be told the truth, but they get nothing, or lies, or partial truths that are not helpful. This is not a problem only in Qatar (many others have done exactly the same), but is definitively made worse by Qatar censorship and heavy-handedness against any "troublemakers."

I learned a couple of things. First, the public is very weak in these situations. You cannot know if your data has been leaked or not, Downloading the archives and verifying yourself takes some technical knowledge that, while superficial, is beyond reach for many people. The are services that can help you, such as Have I Been Pwned?, but they use only e-mail IDs.

There are way too many choke points, from legal systems, to censorship systems, to hosting providers, that can be used to silence anyone. Also, if I had been in Qatar I would have risked jail time followed by deportation, and most likely I wouldn't have done this under those circumstances.

Second, we are also quite strong, for many reasons. The leaked archive is out there and is not going to disappear: you don't get the genie back in the bottle on the internet. That is good because people can still verify if their information was stolen, but bad because the archive contains personal data and hence the potential for abuse is huge.

We are also powerful because we are many and we can move fast, way faster than what many corporations and governments can.

What else did I learn? If I had to do the same thing again, most likely I would find a secondary hosting provider and do it there; the price of getting your main hosting account attacked is too high, and while I was expecting QNB would go to my hosting provider, I thought they would at least look at the complaint to examine its merits. There are many hosting providers that are better suited for this than the standard ones. I would also set-up this under https from the beginning to make it more resilient and secure.

Everything else, I would do the same. It was the right thing to do, and I'm glad I was able to help.

(Pardon my French ;-)

Don't worry, I'm not going to embark in a tirade against religion. While I do believe, as Voltaire remarked, that “those who have the power to make you absurd have the power to make you unjust,” this is beyond religion.

We have been relentlessly led to believe, for decades now, that somehow ideologies do not have a place in politics anymore. Instead, all we should pursuit is a rational approach to practical problems.

Among the stupid things we believe, this is probably the most stupid one.

A political ideology is simply a collection of ideas that is more or less comprehensive, in the sense of covering different aspects of our social life. Indeed, each one of us lives in a society that is essentially kept together and driven by a political ideology, something we fail to notice until the ideology changes or we experience a different society. It is like the smell of the city we live in, something we don't notice until we come back from a long trip.

The ideologies we live in are made of many ideas, some of them good, some of them bad. Saying that political ideologies are dead is just an attempt to convince us that we shouldn't revise the ideas that drive our particular society at a particular moment, because they have somehow proven to be correct.

I don’t think that is the case. We have been wrong about lots of things in the past, even in the recent past, and most likely we continue being wrong about a whole lot of other things, right now.

The ideologies that we live in are very important in politics because they determine which proposals for change are considered seriously, and which are summarily dismissed. In the political arena, ideologies determine who is reasonable and who is insane.

By accepting, against all rational thought, that the particular political ideology in which we live is somehow optimal, we have decided that we don't want to hear anything that challenges it. This fossilizes deeply held but ultimately stupid beliefs, including:

1. That the greed of others is good for us.
2. That the next war will make us safe.
3. That politics and politicians are bad.
4. That something is always going to save us, -or-
   
   That it is best to sit and wait for a collapse.

Abandoning these and other stupid beliefs won't solve everything: one can take great ideas and great intentions, and do something awful with them. However, many of the worst decisions we've collectively taken during 2015 (and accepting them passively is part of that) ultimately can be traced to some of these bad ideas we haven't been able to revise.

We believe the greed of others is good for us

This is a faulty generalization from the observation that, under specific circumstances, specific types of greed can create specific social benefits. For instance, when Adam Smith coined his famous “invisible hand” phrase, he was referring specifically to the preference of investors for domestic investment instead of foreign one. Instead, we have discarded the contexts in which greed might be good, and take this as if it where some universal law of nature.

Believing that the greed of others is good for us has allowed entire industrial sectors to capture the regulators that are suppose to keep them from harming us (and themselves!). Greed in our political system means economic interests determine what changes and what stays the same, while the interests of citizens have little influence.

Even worse, the defense of unrestricted greed has tied it to concepts that have nothing to do with it. For instance, placing limits on greed doesn't mean we don't respect private property. Instead, it means we want the property of the poor to be protected, not only the property of the rich. To me this has nothing to do with ending capitalism. But maybe I'm wrong, and, as Slavoj Žižek has repeatedly warned, I find it easier to imagine the end of the world than the end of capitalism.

We believe the next war will make us safe

In the US, heavy furnitures such as unstable bookshelves and large TV sets crush to death about 30 people every year, many of them children. Preventing their deaths would not require expensive military campaigns, yet the "war on terror" seems to be exactly what the US public think it needs in order to feel safe.

In France, after the Paris attacks the president asks and obtains “special powers” than he now wants to make permanent through constitutional amendments. War mongering leaders in Europe and America play with fear and swear they will protect us … if we just give them a little more power.

They say they will protect us by starting and winning a war. At the end of the war, there will be celebrations with music and fireworks, and everyone who was against us will surrender and quietly go home while the credits roll. They will promise never to harm us again. The world will be at peace.

Except it won't. It never has.

Peace is difficult to achieve, and killing people is a quite cinematic but fairly ineffective way of progressing towards that goal. Peace has many pre-conditions including an effective government, low levels of corruption, a sound business environment, acceptance of the rights of others, and high levels of human capital. Too many of us believe safety will be achieved through more wars; too few of us are thinking on how to create peace.

We believe politics and politicians are bad

Contrary to popular belief, our politicians don't fall from the sky. They are born among us, and while they often come from wealthy families, in the end it is we who vote for them. Do we have the ones we deserve? Maybe.

Politics is a way of distributing power and it plays a role in revising ideologies. Neither politics nor politicians are inherently bad. They are part of a process.

The problem is the process we have right now creates strong incentives for politicians to focus on two things, none of which is good for us. First, politicians have incentives to publicly and vocally support one or two policies favored by undecided voters, not majorities. Second, politicians have incentives to privately and quietly support whatever favors the elites who can help them get elected in their next campaign.

These things won't change overnight, but leaving democracy to its own devices will hardly make anything to improve it, unless …

We believe something will save us in the end -or-
Waiting for a collapse is the best

Mainstream left- and right-wing politicians rarely agree on something. When they agree on something, it is usually along the lines of some corporate interest. In the case of global warming, the left believes it is a serious problem, the right believes it is not, and the consensus is that it is a serious problem but we should do nothing about it.

The underlying belief is that something will save us in the end. Yes, the temperature will rise a few degrees, some polar bears will have to get a job in the circus, and a few small islands will be lost to the rising seas, nothing to worry about. Someone will invent an app or a machine that will make greenhouses go away, or perhaps others will change their habits so we won't need to.

This is indeed stupid; particularly considering the price tag of this stupidity might be astronomically high.

Some believe this is part of an impending collapse that ultimately will be good for us. I am not speaking about judgment day as, among others, extremist Christians and Muslims expect to happen any time soon.

Instead, I am speaking of a “rational” strategy, which is as follows. First, we completely refrain from political participation so that governments become increasingly illegitimate. Second, as we withdraw to the fringes of the system, we let a few people control most of the resources and take all the decisions. Third, we allow conditions in the planet to deteriorate to the point where things are unbearable and people start to die. Fourth, we chop a few heads, rename the months of the year, and start over.

Great plan—where do I click to support it?

It is easy to forget that we're a primitive society

As we celebrate progress, it is easy to forget the obvious fact that we live in a fairly primitive society. We stand divided, in more than one sense. Most of us can only communicate with a fraction of our fellow humans. None of us has ever left the close vicinity of our home.

“Maybe you earn less than your parents because you don’t have ideologies” said journalist Antonio Baños to masses of unemployed and underemployed voters in an interview.

We do have ideologies, plenty of them, the problem is that we don't recognize them as such, we take them as given, we tiptoe around them, we refuse to question them. We should understand they are opinions, not facts, and—to quote Voltaire once more—“opinions have caused more ills than the plague or earthquakes on this little globe of ours.”

I love research environments, even when they are far from perfect. There are a few bad people, plus a great deal of good people that occasionally do bad things because they are single-minded or focused on their own goals. However, the bad things are a small part of what a research life has been to me.

It's like Barcelona: it's a great city, but there are pickpockets. I don't let that distract me from the fact that it's a great city. That doesn't mean I think they are not bad, nor that I endorse what they do, nor that I think they are some sort of necessary evil. It just means I don't let them define my experience of the city.

To enjoy a research life, one needs to learn to focus on what matters, to speak up when needed, to not take things personally, and to forgive and forget. Dwelling on the bad just ruins the experience.

For centuries we've designed complex systems through a superposition of layers. The outer layer, the "user interface," is in direct contact with users. The inner layers correspond to subsystems that work on behalf of the users, but with which users have little or no direct interactions.

Everything is designed that way, not only software and gadgets:

• Food is a complex system in which the user interface includes package labels, supermarket shelves, and restaurant menus.
• Democracy is a complex system in which the user interface includes voting ballots, media coverage of politics, and demonstrations.
• Banking is a complex system in which the user interface includes automatic and human tellers, point of sale systems, and stock tickers.
• Healthcare is a complex system in which the user interface includes waiting rooms, nurses, doctors, and pharmacists.
• Cars are complex systems in which the user interface includes the steering wheel and the dashboard.

We use many metaphors to speak about the inner layers. We speak of what happens in the "back office" or "under the hood." We assume whatever happens there doesn't concern us as long as the job gets done. System designers assume, in turn, that most of the time users don't need to know about the inner layers. Indeed, when an inner layer is inadvertently exposed through the interface we see this as a system failure; sometimes a benign one such as a bit of dirt in supermarket vegetables, sometimes a creepy one such as a chicken head in a bag of McNuggets.

All systems require transparency to use them effectively, to recover from failure, and to build upon them.

Despite what naïve system designers may think, in most systems users often need to know about what internal layers are doing: to use a system effectively, to recover from failure, and to build upon it or customize it to their needs. For instance:

• A healthy diet requires knowing what goes into our food. A sustainable diet requires knowing where does it come from and how it is prepared.
• Democracy cannot be construed to mean blindly voting every few years, and actual progress requires understanding how legislative change works.
• Reducing the harm from the next economic crisis, requires understanding what can each of us should do to prevent it from being too deep.
• Maintaining good health and recovering from illness without bankrupting hospitals, increasingly requires relying of patients' self-care.
• Fuel economy and safe driving requires a basic understanding of how a car engine works, and what are the symptoms of possible failure.

Some of the biggest crises we're facing today are caused or aggravated by the fact that we're hiding too much information in the name of "simplicity." We're increasingly becoming separated of sub-systems that are vital for us. Complete opacity is bad design, sometimes intentionally so. Usable systems have a transparent outer layer and ways of interacting with inner layers progressively.

Power structures that predate the information age deal with opacity by creating more opacity.

The way in which this information problem is addressed, and sometimes ignored, mirrors pre-information age power structures. Indeed, they tend to encourage more opacity: instead of transparency and access to information, we see more and more layers of specialists, representatives and regulators who are supposed to represent our interests and keep us safe:

• Food production companies fight tooth and nail against any initiative to expand information to consumers. Instead, they encourage industry-led committees that determine what is good for us.
• Instead of implementing transparency by default, most governments implement transparency on request, often behind a bureaucratic maze.
• Banking oversight as implemented now requires us to blindly trust on the same people who repeatedly failed to prevent banking crises.
• The only information we get about pharmaceuticals are advertisings encouraging us to use the newest medications.
• Car diagnostics are hidden through proprietary systems that make (self-)maintenance impossible or artificially more expensive.

Professionals that create computing and information processing systems have our share of responsibility on this. Not only we design big parts of these systems, we create the wrong metaphors than shape the industries and the expectations of users.

We're encouraging users to believe that ignorance is a good thing.

Most notably, we're encouraging people to believe that ignorance is a good thing. Systems where users are "taken care of" and "don't need to worry about anything" should arise suspicion, not praise. As the world becomes more complex and people have a proportionally vanishing understanding of what happens around them, we should enable and encourage exploration.

A possible framework to achieve this is what Jonathan Zittrain calls generative systems, but there is much more to it. The starting point, in my opinion, is to realize that an entire generation is actively being prevented from understanding the systems around them. This is a huge step backwards. People learn about the systems around them by using them: let's encourage and enable that learning.