VODA

Water level measurement project

In the "water" project, as the main database programmer, I was in charge of the development of a web application about the quantity and quality of water streams. The development took place in the Postgres database environment on the Apache Superset frontend. The project, due to the complex set of data and the higher number of involved entities, was from the beginning specific and presented a challenge. Considering my competitive nature, I took it as an opportunity to improve and acquire new knowledge from the world of open source technologies or to get to know the principle of operation of smaller companies and projects of the European Union. The specificity of the project is also said by the fact that at the beginning I joined as the only database programmer and at the end there were up to 10 of us. The main goal of the project was to unify previously scattered applications into one without future waste of state resources. My task was to transform the data from the original Oracle database into Postgres and to find a way to design reporting in an optimized and comprehensible way, so that the response of the database and the application was not long, and that it met the strictest criteria, namely scalability and future sustainability. Another need was to adapt reporting to the Apache superset, which hindered the principle rather than helped it. At the beginning, we designed the database according to the postgres optimizer and downloaded data from Oracle via datapump. Subsequently, the structures of new tables were created and data for reporting was poured into them. Subsequently, reports began to be created. For reporting purposes, we chose functions with input parameters with predefined values, which returned an SQL query at the output. Compared to tabular and materialized views, which are commonly used for reporting, this solution was the most effective. After devising a processing method, we received a number of new reports from the customer, which were not in Oracle before, and thus the need to hire new database managers increased so that the project could be delivered within a year. Due to the lack of experienced workers on the market, I was forced to train testers, java programmers or analysts in writing SQL scripts. This created the need to do training, which you can find in the Blog section. In addition to the reports themselves, we discovered that the data structure will change during the project, so we devised an etl process that transformed the data from new structures to the structures needed for the reports. In the meantime, several rounds of migration from an external supplier took place, after which we tuned the transformation processes. Thanks to the spirit of the company's management and loyal employees, it was possible to hand over the project on time with comments.

Benefit for the company

The project brought me knowledge about new open source technologies and proved that if there is good project management and a joint team setting, things can be completed even if it doesn't look like it. It also showed that even people with less knowledge of the database world can help if they will be given the necessary training.