Managing Technical Debt

Martin Fowler has a great explanation of this term and the problem behind it. Many startups and new projects use quick-and-dirty solutions to achieve their goals as soon as possible, which overcomplicates a project, slows it down, makes it hard to change or implement new features and full of bugs. I’d like to bring up the most common issues and how to manage them in the most effective way, from my point of view.

A project with big technical debt leading to the fear of change. Any change can have unexpected effect and produce more bugs. And there is only one way to fearless development - it’s automated tests, which usually don’t exist on such projects. So technical debt is mostly about luck of control of the project quality.

There are other common issues on the projects with big TD, but this is the most important and, unfortunately, a heavily underestimated one.

Measuring software engineering competency

Why do we want to measure our competency? Well… because we want to improve. And we have to! The world is rapidly changing: tools, technologies, paradigms, concepts, processes, competitors, crises. We must run to stay in business. We must deliver the product, maintain the quality, meet the budget… But competency and quality measuring is tough! We don’t have rules to assess them. Joel Spolsky tried to build a simple checklist for a team to rate its quality in his famous The Joel Test, but it was sixteen years ago. So let’s stand on the shoulders of giants, and revise it from just the test to a set of rules that can guide us for the better software engineering competency.

Allow to send HTTP/2 requests to APNS on CentOS6 and Python3.5.

Trying to send Apple push notifications (APN) on CentOS6 using python hyper library got the ConnectionResetError And it looked like hyper is trying to use HTTP1.1 instead of HTTP/2.

It turns out hyper requires an ALPN (Application-Layer Protocol Negotiation) to negotiate HTTP/2. The latest openssl version available for eh6 is OpenSSL 1.0.1e, but ALPN was introduced > 1.0.2. In order not to brake dependencies, we need to compile and install new openssl library into our custom location. Also we need to compile Python specifying custom openssl library.

Let’d do it…

BDD tests for Hadoop with Cucumber. Part II

In previous part we have created the application to process the data and docker container to isolate testing environment.

In this part we will create tests using Cucumber for Java and Gherkin.

Acceptance tests, usually, check the normal behaviour of application - we want to be sure that the app do what is designed to do.
Gherkin is the great way to create acceptance tests - because it allows to concentrate on what we are going to do instead of how we do it, and, as a side effect, we get document features and scenarios in human-readable format which is very helpful, especially for new team members.

BDD tests for Hadoop with Cucumber. Part I

Big Data is a processing of a huge amount of structured (more or less) data that should be filtered or sorted for future analysis or, actually made analysis. So much data that you can’t process it on the one computer for a reasonable time. This can be logs from services, like call service, or logs from web servers that contain billions of records, so you should process data in parallel on a bunch of computers. To do it so we use specific software based on MapReduce pattern. In our case, we use Hadoop with Cascading framework. Hadoop implements MapReduce and Cascading framework contains a lot of useful tools, for example abstraction to Amazon S3, so we can use S3 buckets as input/output folders for processing jobs, but at the same time we allow to use regular file systems on developer machines for development and test purposes.

As any other piece of software it should be tested before using in production, especially because of high cost of logical mistakes in processing a lot amount of data. Unfortunately there are no much information about BDD testing of Hadoop and Hive jobs. So I decided to write how we do it in our Cascading project here, in Intelliarts.