Data Lake Platform At Viant, the cloud ad server stack produces 1.7 M transaction log files, resulting in 70 billion records taking up with 80 TB of data daily, where each server rotates log files every 3 minutes and uploads them to Cloud Storage. Our goal was to ingest all data log under…

In the previous article, I’ve discussed automation techniques leveraging security and developer-centric multi-stack build and deployments. In this one, I will focus on an application state setup automation. Before deep diving, let first define what the application state is. Application state is data driving your application logic,which may originate in…

Developing software requires many repetitive tasks. No matter if you rebuild/redeploy an app, restart services or reset an application state, all these tasks add up a substantial amount of non-productive time. There is no secret that the best way to improve productivity is automation. For example, using docker to build…

Storing raw level data in a database becomes increasingly important. Not only can it be a source for data analytics, but also it can be crunched by a data scientists in any way it serves the business. …

Go lang tries to distance itself from object-oriented terminology, at the same time it can give you all object-oriented programming flavors and benefits. For instance, C++ or Java classes are called a struct, A class methods are called receivers. Inheritance is called embedding In this post, however, I will focus…

Many of us faced the challenge of data handling during integration and end-to-end test automation. In this article, I look into various data organization patterns, data loading strategies and data validation requirements. A typical use case may require ingesting data to a dozen tables; also the data can be a…

Have you ever found developing end to end testing for your ETL application challenging or difficult, not anymore, this article presents a practical walkthrough of various tools and technology leveraging testing complexity. End to end testing End to end testing (e2e) is the most comprehensive testing methodology which tests entire application in an environment…