Local vs. Remote Port Forwarding

Have you ever wondered what port forwarding is or what the difference between local and remote port forwarding is? Port forwarding is a way to forward or “tunnel” TCP traffic through SSH from one machine to another. Using just one line of code, you can create an outgoing tunnel, forward your IP requests over that tunnel, and receive the response. In this way you can pull the data from a remote server to a local server (local port forwarding), and your local machine acts as a proxy server for the remote one. Or you can create an incoming tunnel to a remote server which receives IP requests, forwards them over that tunnel to the local server where it is processed and sent back again. Thus it is possible to push data from a local server to/through a remote server (remote port forwarding).

Local Port Forwarding (Outgoing Tunnel):

  • Principle: Local host forwards/displays content of remote host. Local host acts as proxy. Tunneling opens a listening socket on localhost and transfers content to remote server
  • Command: ssh -L local_port:remote_host:remote_port login@servername
  • Tunnel: local host -(SSH tunnel)→ remote host -(SSH tunnel)→ local host
  • Example: check remote host behind load-balancer or firewall on localhost

Remote Port Forwarding (Incoming Tunnel):

  • Principle: remote host forwards content of localhost. Remote host acts as proxy. Tunneling opens a listening socket on the remote server host and transfers the content to the local host
  • Command: ssh -R remote_port:local_host:local_port login@servername
  • Tunnel: remote host -(SSH tunnel)→ local host -(SSH tunnel)→ remote host
  • Example: make localhost visible in the internet or giving access to a service on your home machine to people at work

Sisyphus Projects

You know death march projects: in the software development and software engineering industries, a death march is a name for a project that is destined to fail. Some software projects are Sisyphus projects: you build it up only to break it down a bit later. An example are Facebook applications: you build them up in a cumbersome process, and if you are finally ready, you can break it down and start rolling up the boulder again, because Facebook has changed its API again. The old API is now deprecated, and the application is no longer compatible with the new one. Congratulations! One feels a bit like Sisyphus who rolls the boulder up the hill. Remember in Greek mythology Sisyphus was a king punished by being compelled to roll an immense boulder up a hill, only to watch it roll back down, and to repeat this action forever.

Optimizing web applications for different browsers can be cumbersome, too: if you finally have optimized you application for IE6-9, and you have fixed the 3 Pixel Jog Bug and other nice relatives, there is a new browser IE 10, followed rapidly by IE 11-19, which behave completely different and introduce bugs you never dreamed of. You can start all again rolling the boulder up the hill. Sometimes software development can be exciting and fascinating, but sometimes it is just frustrating and exhausting.

Some social networks are like black holes for data

In earlier posts I tried to compare start-ups and quasars, and social networks with expanding universes. Now I would like to take a look a “black holes” and their relation to social networks.

From the inside social networks are like expanding universes, they grow bigger each day by adding more members, more relationships, and more informations. From the outside they are quite the opposite: they are acting like big black holes who suck in information and make it inaccessible for everyone outside. Robert Scoble compared large closed corporate social networks like Facebook to black holes and coined the term “data black holes”. Data vanishes inside these black holes, and never comes out again. Can we save the common, open web or is it too late?

Currently, Facebook behaves indeed much like a data black hole who sucks in a larger part of the web and private information every day. Facebook is much more closed in this respect than Google+, which has like Twitter a public feed, better private/public sharing options through the “circle” feature, and it allows the export of data by “Data Liberation”. After the IPO, Facebook wants to grow and expand its own intergalactic advertising universe even further. Whether this will be successful or not will be seen. Will Facebook be able to beat Google AdWords with their own Advertising programm by sucking in the majority of information and traffic of the web? What do you think?

(The image of a quasar or growing black hole is from Wikipeda and can be found here)

Programming together

Programming together in a group is a bit like living together in the same house or sharing the same apartment. It can be fun if all work together. It can be frustrating and annoying, too. Others will get upset if you don’t clean up the kitchen (and do not leave clean code), mess things up or if you change their things. In an apartment- or flat-sharing community, it is often the shared rooms like kitchen and bathrooms that cause the most trouble.

To be more precise, software engineers or software architects who work together are a bit like architects who live together in the house they are building. It works best if everybody agrees on a central plan (if there is any) or if everyone works and “lives” in his own space, i.e. everyone has his own room(s), owned only by himself, where the others are not allowed to change things. If you are lucky, then the others may show you their rooms occasionally. Some shared regions like the main entrance, can be public, though. Basically, programming well with others is difficult, just like living together, because likes, tastes and preferences vary.

Since programmers are like little gods, almighty and all knowing entities in the world they have created, the clash begins already if two or more programmers with equal rights must work together. One wants a red room, the other a green. One wants wooden furniture, the other plastic. To find a compromise means to communicate and talk with each other. And this is an area were most programmers do not excel. I am not good at it, either. I think it is justified to say many have chosen to become a programmer in order to avoid too much idle talk and communication. This is were the difficulties begin..

(The picture is from Flickr user Annette Bouvain and shows a flat share)

A minimum number of chips and code

Good hardware engineers like the legendary Steve Wozniak make designs which have a minimum number of chips. They try to design a simple minimum-chip circuit, because in hardware design a minimal number of chips means to minimize the cost.

Good software engineers make code which have a minimum number of lines. They try to design a simple minimum-code program, because in software design a minimal number of lines means to minimize the complexity. Less code means less effort to understand it, less trouble while maintaining it, and less hassle in changing it.

Thus in general, less items in engineering means less cost and less complexity. A minimum number of items, whether chips or lines of code, is important to achieve simplicity and the KISS principle. The KISS principle states that most systems work best if they are kept simple rather than made complex, therefore simplicity should be a key goal in design and unnecessary complexity should be avoided.

(the image from Flickr User Fellowship of the Rich shows an Apple motherboard)

Social networks are like expanding universes

The universe is not static, it is expanding ever since the Big Bang 13.7 billion years ago. It contains billions and billions of stars and galaxies, and it is getting bigger all the time.

Social networks like Google+, Twitter, or Facebook are similar. They do not only contain billions and billions of users and pages, they are also getting bigger all the time. Each day, more users join, and existing users add more content: they add more photos, share more links, and write more posts.

This is the major challenge in the engineering of social networks: to handle a system which is getting bigger all the time. An engineer of such a network is like a cook whose pots are constantly boiling over. He must add more pots each day, and keep the old still cooking.

Facebook was founded in 2004, and it was expanding ever since, gaining more users and more friends each day. Since it does not allow users to quit, and does not delete old content, it grows more and more each day. Twitter was founded in march 2006, and it was expanding ever since that time, too. Each day more users join, existing users gain more followers, and create more tweets. The latest and most advanced social network is Google+, which was started in July 2011 and combines the best features of Facebook and Twitter. It is growing at an ever accelerating rate, too.

The fundamental problem for the engineer is clear: a system which is getting bigger all the time is hard to handle, because it requires nearly unlimited scalability. If the engineers do not consider this from the start, it will be hard to build it later. Apparently, Google has the best prerequisites to master this challenge, it knows how to build scalable systems better than any other company. Some websites say Google is the King of scalability.

(The picture of the accelerating universe is from Wikipedia)

Programming well with others

Recently there was a post at the 37signals blog Signal vs. Noise about the question why programs become territorial. The reason is simple: Programs become territorial because programmers naturally consider their code as their territory. Your code is your work and your creation based on your ideas, right? Therefore as a programmer it is natural to think that your code belongs to you. It is the world you have created. And in his own world the programmer is a little god who knows everything, while everyone else is a mere mortal without superpowers. The own world contains only cities and places the programmer has built himself – so the programmer is the creator, architect, ruler, general, and governor all in one. A programmer which tries to understand the code from someone else has not only lost his superpowers, he is completely lost. It is a bit like being a tourist in a foreign city – you will feel lost in the beginning. Maps in form of database diagrams and UML models help to a certain degree, but even with a map it remains difficult to understand a foreign territory.

Software engineers tend to own their creations. It goes well in two cases: if everybody knows and owns his own part, i.e. if everybody has his own territory, or – if if the territory belongs to the team. This means everybody knows and can change everything, everything is well documented and the team agrees on certain coding standards, policies and guidelines. Unfortunately, working in a team is not easy, because it is difficult to create useful documentation, and programmers do not like giving up their freedom. They like their superpowers. Developers want autonomy, mastery and purpose. Working in a team is difficult, because it means to have a team ego instead of an individual ego. Agreeing on standards does not mean that everyone in the team has to accept tools, libraries and plugins you prefer, rather it means to accept things others prefer more than you, and to abandon personal habits and likes. This is not easy, because programmers are often very opinionated. So the two extremes that work are:

  • The individual rules: team has no strong ego, territory belongs to the individual
  • The team rules: individuals give up their ego, territory belongs to the team

Between those extremes it gets complicated. Code changes can lead easily to personal conflicts if code is owned by people. Therefore developers tend to avoid changing the code from others. Programming well with others is difficult. It certainly requires good communication. Most programmers know how to communicate, but they don’t want to. They want to do it themselves, and they see others as competitors. It takes indeed a bit the fun out of programming if you have to justify and explain every decision to others. To find the right balance between cooperation and competition in a team is not easy. In general, the golden rule helps: One should treat others as one would like others to treat oneself. Do not ignore them, ask others for their opinion, show them your respect, include them in important decisions, explain your architecture, show them your achievements etc. Communication is the glue that keeps groups together. This is the reason why most programmers have difficulties when it comes to programming well with others. They lack social skills, and the only languages they really like are programming languages. They are used to work with machines and not with people.

Ben Collins-Sussman and Brian Fitzpatrick give us some hints how to program well with others. Documentation is important: document your design decisions, your architecture, your goals, your failures, for example on an easy editable Wiki. The ability to communicate is also an important aspect. They argue in this video (and their forthcoming book with the same title) that collaborating and communicating with others is at least as important as having great technical skills. According to Ben and Fitz, the right attitude is important in a team. You should be thinking about “how do I work with a team” and not “how do I present myself as a genius”